- Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] before: if docker images not exist, not build images after: if docker images not exist, build images Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..a60067f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wu Fengguang 14 Jan '21

14 Jan '21

>>>+ echo "api-avx2neon running" >>>+ ./test >> >>What if ./test exit 1? > > this script captures and counts its own error output What if ./test exit for other (unexpected) reason? Then we miss a failure case. >>>+ echo "api-avx2neon finished"

1 0

[PATCH v3 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress.build.fail: 1 [output] wordpress.build.fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..e31934573 --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build.fail/ + + puts line +end -- 2.23.0

2 1

[PATCH compass-ci] qeumu/kvm.sh: modify the comment
by Hu Xuejiao 14 Jan '21

14 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index ce3b75f..8b05f83 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "The kernel does not exist" + echo "The kernel does not exist: $kernel" exit } -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress_build_on_fail: 1 [output] wordpress_build_on_fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..3f8012a8c --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build_on_fail/ + + puts line +end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

TO ALL. On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. It does not happen if you didn't say it. That's the golden rule of mailing list. Please document such important points in the changelog in future. Thanks, Fengguang

1 0

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. > > if I create new file under the dir would not cause this error. > so i think there is no need to upload empty folder at all. No "think" please. You should give clear statement, like "webdav supports auto-and-recursively create dirs when uploading some dir_level1/dir_level2/file". And prove it with commandline examples or webdav API definitions. Whether or not your patch is right, the changelog is not clear enough to prove whether it's "definitely/always right" or "right by chance". >> >> mkdir a => success >> mkdir a => 409 >> >>But will the server side auto create new subdirs if they do not >>already exist? Probably the below find will fail. > >> >>Please experiment with deeper dirs, like a/b/c/file > > Actually, I have experimented with deeper dirs Then please showcase it in changelog. BTW, please try this example https://stackoverflow.com/questions/42941176/how-to-create-directories-recu… If our webdav server can auto create subdirs, then what's the difference in our server comparing to the standard/old one? Thanks, Fengguang

1 0

[PATCH compass-ci] lib/error_messages.rb: uniform naming style
by Lin Jiaxin 14 Jan '21

14 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 14 +++++++------- lib/git_bisect.rb | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..3e1564b 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -8,8 +8,8 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # usage: # foo = ErrorMessages.new(build-pkg) # build-pkg is a file that includes errors generated by make -# errorid_array = foo.obtain_error_messages_by_errorid(errorid) -# errorid_array includes all error_messages that error same as errorid +# error_id_array = foo.obtain_error_messages_by_error_id(error_id) +# error_id_array includes all error_messages that error same as error_id # error_messages_hash = foo.obtain_error_messages # error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } # output: @@ -40,15 +40,15 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_errorid(errorid) - error_messages_by_errorid = [] + def obtain_error_messages_by_error_id(error_id) + error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == errorid - error_messages_by_errorid += v.to_a + if ('build-pkg.' + build_pkg_error_id(k)) == error_id + error_messages_by_error_id += v.to_a end end - error_messages_by_errorid + error_messages_by_error_id end private diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 2043730..2fee60c 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -97,7 +97,7 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) -- 2.23.0

1 0

Re: [PATCH lkp-tests] lib/job.sh: modify the generated file suffix
by Cao Xueliang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 03:44:10PM +0800, Li Ping wrote: >[why] >user extract $program result actually generating a YAML file. >and compass-ci/sbin/result2stats have already support merge yaml file > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > lib/job.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/job.sh b/lib/job.sh >index 03c49e0ad..dc65f68d8 100755 >--- a/lib/job.sh >+++ b/lib/job.sh >@@ -318,5 +318,5 @@ run_test() > run_target_stats() > { > local script_name=$1 >- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml I don't think this can change json to yaml format. Thanks, Xueliang > } >-- >2.23.0 >

1 0

[PATCH v9 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: reduce possible duplicate judgment
by Li Yuanchao 14 Jan '21

14 Jan '21

The method is called several times, add judgment in the method, so don't need to add it everywhere the method is called. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2adaaa1..86f02e5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -187,6 +187,8 @@ class MirrorMain end def do_push(fork_key) + return if @fork_stat[fork_key][:queued] + @fork_stat[fork_key][:queued] = true @git_info[fork_key][:cur_refs] = get_cur_refs(fork_key) if @git_info[fork_key][:cur_refs].nil? @git_queue.push(@git_info[fork_key]) @@ -196,7 +198,7 @@ class MirrorMain return if @git_queue.size >= 1 fork_key = @priority_queue.delete_min_return_key - do_push(fork_key) unless @fork_stat[fork_key][:queued] + do_push(fork_key) @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) @priority += 1 end -- 2.23.0

1 0

[PATCH v4 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..99aeccc1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..5305472 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,41 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +84,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +96,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..8348554 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,42 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + puts "keyword_list ==> #{keyword_list}" + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +85,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +97,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 1

[PATCH compass-ci] container/add-software: refactor "cat <<-EOF" to make code aligned
by Xu Xijian 14 Jan '21

14 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/add-software/run b/container/add-software/run index 840fb2c..aab659b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. [ -L "$1/boot/vmlinuz" -a -n "$2" ] || { - cat << 'EOF' + cat <<-'EOF' Example usage: 1> ./run /os/centos/aarch64/7.6.1810 pack1 pack2 ... 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) @@ -11,7 +11,7 @@ the whole path is '$LKP_SRC/distro/adaptation/centos'. [Remind] This tool also works for openEuler. -EOF + EOF exit } -- 2.23.0

1 0

[PATCH v8 compass-ci 1/5] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..6003028 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,10 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

2 4

[PATCH lkp-tests] daemon/sshd: refactor sshd
by Wei Jihui 14 Jan '21

14 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/sshd | 61 +++++++----------------------------------------- lib/email.sh | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 53 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index e9f38006..7ba471a8 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -10,6 +10,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/http.sh . $PROJECT_DIR/lib/env.sh +. $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 @@ -36,56 +37,6 @@ run_ssh() fi } -data_success() -{ - data="To: $my_email -Subject: [NOTIFY Compass-CI] Machine application successful - -Dear $my_username, - - Your test machine has been provisioned. - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - -Regards -Compass-CI -" -} - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application failed - -Dear $my_username: - - Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. - -Regards -Compass-Ci -" -} - -compose_email() -{ - deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n "$port" ]; then - data_success - else - data_failure - fi -} - set_port() { [ -n "$my_ssh_pubkey" ] || return @@ -103,9 +54,13 @@ set_port() run_email() { [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" + + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + send_email "borrow_success" + else + send_email "borrow_failed" + fi } run_ssh diff --git a/lib/email.sh b/lib/email.sh index d7c2c44a..ed75e0ce 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -15,6 +15,72 @@ Compass-CI " } +job_debug_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job debug: $id + +Dear $my_name, + + Your test job is ready to debug and test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + + +borrow_success_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application successful: $id + +Dear $my_name, + + Your test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application failed: $id + +Dear $my_name: + + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-CI +" +} + send_email() { local subject=$1 -- 2.23.0

1 0

[PATCH v4 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

before: docker images exist, then build images after: docker images not exist, then build images [error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] when docker images not exist, will not execute ./build, so failed to build images. [how] when docker images not exist, then execute ./build. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..77fe43a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH v6 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 14 Jan '21

14 Jan '21

before: network interfaces "up"|"1" -> network is ok after: network interfaces "up"|"1" && export_ip_mac ok -> network is ok [error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= even network interfaces ready, but routing not, get ip by routing failed, so add a check export_ip_mac to ensure network is ok. test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351368 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351388 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351369 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

2 2

[PATCH compass-ci 2/2] src/delimiter: delete unused requires
by Cao Xueliang 14 Jan '21

14 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/delimiter.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 1afffbc..a66f4fb 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -5,9 +5,7 @@ require 'json' require_relative './constants' -require_relative '../../lib/git_bisect' require_relative '../../lib/taskqueue_client' -require_relative '../../lib/mail_bisect_result' # consume assister task queue class Delimiter -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/delimiter: no need mount "/srv/git" and delete some ENV vars
by Cao Xueliang 14 Jan '21

14 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 8 -------- 1 file changed, 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index cb44d84..669923b 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -7,15 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - ES_HOST - ES_PORT - SEND_MAIL_HOST - SEND_MAIL_PORT - MONITOR_HOST - MONITOR_PORT TASKQUEUE_HOST TASKQUEUE_PORT - GIT_MIRROR_HOST ] defaults = relevant_defaults(names) @@ -38,7 +31,6 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro - -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter ] -- 2.23.0

1 0

[PATCH v8 compass-ci 5/5] send-internet-mail: email rate limit
by Luan Shengde 14 Jan '21

14 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value. the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the mail address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 62 ++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 lib/mail-post/email_limit_queue.rb diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/lib/mail-post/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0