- Compass-ci - mailweb.openeuler.org

[PATCH v7 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 14 Jan '21

14 Jan '21

[Error Msg] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [Root Cause] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= [Why] even network interfaces ready, but routing not, get ip by routing failed, so add a check export_ip_mac to ensure network is ok. [how] before: network interfaces "up"|"1" -> network is ok after: network interfaces "up"|"1" && export_ip_mac ok -> network is ok Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

4 8

help star
by Wu Fengguang 14 Jan '21

14 Jan '21

Xueliang is our help star, thanks Xueliang! caoxl@crystal ~/code/todo/bin% ./help-star ../people/*/credit/2021-01-14.md 24.6 Wu Fengguang 16.2 Cao Xueliang 16.1 Yu Chuan 12.2 Luan Shengde 11.8 Wei Jihui 11.7 Liu Shaofei 10.2 Zhang Dewan 8.9 Xiao Shenwei 8.5 Zhang Yale 8.3 Liu Yinsi 7.6 Wu Zhende 7.2 Ren Wen 6.9 Wang Yong 6.5 Lu Weitao 5.2 Lin Jiaxin 5.0 Li Ping 4.7 Bai Jing 4.6 Xu Xijian 4.3 Cui Yili 2.4 Li Yuanchao 2.4 Wang Chenglong 2.2 Du Kaitian 1.3 Lu Kaiyi 1.1 Hu Xuejiao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH v5 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..4c7d07b3 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

3 4

review star
by Wu Fengguang 14 Jan '21

14 Jan '21

Jihui is our review star, thanks Jihui! caoxl@crystal ~/code/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 98 32 52 20 Wei Jihui 71 44 125 19 Wu Fengguang 24 13 23 11 Liu Shaofei 59 16 21 9 Cao Xueliang 12 10 31 8 Yu Chuan 41 10 20 7 Lu Kaiyi 33 7 13 5 Ren Wen 16 7 8 5 Cui Yili 16 6 7 4 Bai Jing 17 3 3 4 Wu Zhende 11 6 10 3 Lin Jiaxin 9 4 7 3 Li Yuanchao 6 4 8 3 Xiao Shenwei 18 6 8 3 Wang Yong 19 6 10 2 Wang Chenglong 14 4 8 2 Lu Weitao 20 4 6 2 Li Ping 29 3 4 2 Xu Xijian 55 4 8 1 Luan Shengde 20 2 2 1 Hu Xuejiao 6 0 0 0 Zhang Dewan 51 1 2 0 Liu Yinsi 30 3 2 0 Zhang Yale

1 0

git count
by Wu Fengguang 14 Jan '21

14 Jan '21

Yinsi is our code star, thanks Yinsi! 12 Liu Yinsi 10 Luan Shengde 9 Lu Kaiyi 8 Hu Xuejiao 5 Cao Xueliang 5 Xu Xijian 5 Ren Wen 4 Lu Weitao 4 Wei Jihui 4 Wu Fengguang 4 Wu Zhende 3 Liu Shaofei 3 Wang Chenglong 3 Li Yuanchao 2 Bai Jing 2 Xiao Shenwei 2 Zhang Yale 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Yu Chuan 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Zhang Dewan 0 Li Ping 0 Shi Zhichao

1 0

[PATCH compass-ci 1/3] sbin/auto_submit: fix failed to submit job in container
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. [how] store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..c121ae0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -6,6 +6,7 @@ require 'bunny' require 'json' require 'yaml' +require_relative '../lib/build_my_info_client' # receive message and auto submit job class AutoSubmit @@ -82,5 +83,15 @@ class AutoSubmit end end +option = { + my_name: 'Auto Submit Robot', + my_email: 'autosubmit@localhost', + lab: 'autosubmit' +} + +# fix verification problems when submit jobs by auto-submit service +build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) +build_my_info.config_my_info + auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

2 2

[PATCH v5 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] before: if docker images not exist, not build images after: if docker images not exist, build images Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..882ef02 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "reboot" ];then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH v6 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] before: if docker images not exist, not build images after: if docker images not exist, build images Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..a60067f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wu Fengguang 14 Jan '21

14 Jan '21

>>>+ echo "api-avx2neon running" >>>+ ./test >> >>What if ./test exit 1? > > this script captures and counts its own error output What if ./test exit for other (unexpected) reason? Then we miss a failure case. >>>+ echo "api-avx2neon finished"

1 0

[PATCH v3 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress.build.fail: 1 [output] wordpress.build.fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..e31934573 --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build.fail/ + + puts line +end -- 2.23.0

2 1

[PATCH compass-ci] qeumu/kvm.sh: modify the comment
by Hu Xuejiao 14 Jan '21

14 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index ce3b75f..8b05f83 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "The kernel does not exist" + echo "The kernel does not exist: $kernel" exit } -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress_build_on_fail: 1 [output] wordpress_build_on_fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..3f8012a8c --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build_on_fail/ + + puts line +end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

TO ALL. On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. It does not happen if you didn't say it. That's the golden rule of mailing list. Please document such important points in the changelog in future. Thanks, Fengguang

1 0

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. > > if I create new file under the dir would not cause this error. > so i think there is no need to upload empty folder at all. No "think" please. You should give clear statement, like "webdav supports auto-and-recursively create dirs when uploading some dir_level1/dir_level2/file". And prove it with commandline examples or webdav API definitions. Whether or not your patch is right, the changelog is not clear enough to prove whether it's "definitely/always right" or "right by chance". >> >> mkdir a => success >> mkdir a => 409 >> >>But will the server side auto create new subdirs if they do not >>already exist? Probably the below find will fail. > >> >>Please experiment with deeper dirs, like a/b/c/file > > Actually, I have experimented with deeper dirs Then please showcase it in changelog. BTW, please try this example https://stackoverflow.com/questions/42941176/how-to-create-directories-recu… If our webdav server can auto create subdirs, then what's the difference in our server comparing to the standard/old one? Thanks, Fengguang

1 0

[PATCH compass-ci] lib/error_messages.rb: uniform naming style
by Lin Jiaxin 14 Jan '21

14 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 14 +++++++------- lib/git_bisect.rb | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..3e1564b 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -8,8 +8,8 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # usage: # foo = ErrorMessages.new(build-pkg) # build-pkg is a file that includes errors generated by make -# errorid_array = foo.obtain_error_messages_by_errorid(errorid) -# errorid_array includes all error_messages that error same as errorid +# error_id_array = foo.obtain_error_messages_by_error_id(error_id) +# error_id_array includes all error_messages that error same as error_id # error_messages_hash = foo.obtain_error_messages # error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } # output: @@ -40,15 +40,15 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_errorid(errorid) - error_messages_by_errorid = [] + def obtain_error_messages_by_error_id(error_id) + error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == errorid - error_messages_by_errorid += v.to_a + if ('build-pkg.' + build_pkg_error_id(k)) == error_id + error_messages_by_error_id += v.to_a end end - error_messages_by_errorid + error_messages_by_error_id end private diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 2043730..2fee60c 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -97,7 +97,7 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) -- 2.23.0

1 0

Re: [PATCH lkp-tests] lib/job.sh: modify the generated file suffix
by Cao Xueliang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 03:44:10PM +0800, Li Ping wrote: >[why] >user extract $program result actually generating a YAML file. >and compass-ci/sbin/result2stats have already support merge yaml file > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > lib/job.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/job.sh b/lib/job.sh >index 03c49e0ad..dc65f68d8 100755 >--- a/lib/job.sh >+++ b/lib/job.sh >@@ -318,5 +318,5 @@ run_test() > run_target_stats() > { > local script_name=$1 >- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml I don't think this can change json to yaml format. Thanks, Xueliang > } >-- >2.23.0 >

1 0

[PATCH v9 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: reduce possible duplicate judgment
by Li Yuanchao 14 Jan '21

14 Jan '21

The method is called several times, add judgment in the method, so don't need to add it everywhere the method is called. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2adaaa1..86f02e5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -187,6 +187,8 @@ class MirrorMain end def do_push(fork_key) + return if @fork_stat[fork_key][:queued] + @fork_stat[fork_key][:queued] = true @git_info[fork_key][:cur_refs] = get_cur_refs(fork_key) if @git_info[fork_key][:cur_refs].nil? @git_queue.push(@git_info[fork_key]) @@ -196,7 +198,7 @@ class MirrorMain return if @git_queue.size >= 1 fork_key = @priority_queue.delete_min_return_key - do_push(fork_key) unless @fork_stat[fork_key][:queued] + do_push(fork_key) @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) @priority += 1 end -- 2.23.0

1 0

[PATCH v4 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..99aeccc1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..5305472 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,41 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +84,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +96,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..8348554 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,42 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + puts "keyword_list ==> #{keyword_list}" + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +85,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +97,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 1

[PATCH compass-ci] container/add-software: refactor "cat <<-EOF" to make code aligned
by Xu Xijian 14 Jan '21

14 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/add-software/run b/container/add-software/run index 840fb2c..aab659b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. [ -L "$1/boot/vmlinuz" -a -n "$2" ] || { - cat << 'EOF' + cat <<-'EOF' Example usage: 1> ./run /os/centos/aarch64/7.6.1810 pack1 pack2 ... 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) @@ -11,7 +11,7 @@ the whole path is '$LKP_SRC/distro/adaptation/centos'. [Remind] This tool also works for openEuler. -EOF + EOF exit } -- 2.23.0

1 0