January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v7 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 11 Jan '21

11 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

Re: [PATCH lkp-tests] jobs: and ansible_test.yaml
by Bai Jing 11 Jan '21

11 Jan '21

^^ add Thanks, Baijing Reply-To: In-Reply-To: <20210109124854.6869-1-ylzhangah(a)qq.com> On Sat, Jan 09, 2021 at 08:48:54PM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 9 +++++++++ > 1 file changed, 9 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..9adc4d71 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,9 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ username: #geerlingguy >+ rolename: #jenkins >+ ansible_host: #localhost >+ ansible_user: #root >+ ansible_python_interpreter: #/user/bin/python3 >-- >2.23.0 >

2 1

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 11 Jan '21

11 Jan '21

This script can be invoked to run the ansible command to deploy ansible-roles which you specified in your environment. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..09e5ca9e --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,59 @@ +#!/bin/bash +# username +# rolename +# ansible_host +# ansible_user +# ansible_python_interpreter + +[ -n "$username" ] || echo "username is empty" +[ -n "$rolename" ] || echo "rolename is empty" +[ -n "$ansible_host" ] || $ansible_host=localhost +[ -n "$ansible_user" ] || $ansible_user=root +[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 + +build_mutual_trust() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa + cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys + ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' +} + +install_ansible() +{ + pip3 install ansible + ansible localhost -m ping + #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko + [ $? -eq 0 ] && echo "Ansible is successfully deployed!" +} + +download_ansible_role() +{ + ansible-galaxy install $username.$rolename + [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" +} + +ansible_playbook_run() +{ + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible + cat <<-EOF > "$ansible_path"/hosts + $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter + EOF + [ $? -eq 0 ] && echo "Hostfile has been generated!" + + cat <<-EOF > "$ansible_path"/$rolename.yml + - hosts: $ansible_host + remote_user: $ansible_user + roles: + - role: $username.$rolename + EOF + [ $? -eq 0 ] && echo "Playbook has been generated!" + + ansible-playbook "$ansible_path"/$rolename.yml + [ $? -eq 0 ] && echo "Your playbook is running successfully!" +} + +#build_mutual_trust +install_ansible +download_ansible_role +ansible_playbook_run -- 2.23.0

5 10

[PATCH lkp-tests] sbin/submit: modify job_id to job id
by Hu Xuejiao 11 Jan '21

11 Jan '21

[why] Because we use "es-find (id='')" to search id-information, but when submit, it is (job_id=''). so it is recommended that the value be the same as the ID Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a92556ff..448d6243 100755 --- a/sbin/submit +++ b/sbin/submit @@ -164,10 +164,10 @@ ARGV.each do |jobfile| JSON.parse(messages).each do |msg| if msg['message'].empty? job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job_id=#{msg['job_id']}") + puts("submit #{jobfile}, got job id=#{msg['job_id']}") else opt_monitor = false - puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") end end end -- 2.23.0

2 1

[PATCH lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 10 Jan '21

10 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..9adc4d71 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,9 @@ +suite: ansible_test +category: functional + +ansible_test: + username: #geerlingguy + rolename: #jenkins + ansible_host: #localhost + ansible_user: #root + ansible_python_interpreter: #/user/bin/python3 -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends: add depends for openeuler_docker test
by Bai Jing 09 Jan '21

09 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- distro/depends/openeuler_docker | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/openeuler_docker diff --git a/distro/depends/openeuler_docker b/distro/depends/openeuler_docker new file mode 100644 index 000000000..b31074ad9 --- /dev/null +++ b/distro/depends/openeuler_docker @@ -0,0 +1,3 @@ +docker.io +wget +git -- 2.23.0

1 0

[PATCH compass-ci] qemu/kvm.sh: modify the prompt information of kernel
by Hu Xuejiao 09 Jan '21

09 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..577bca1 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -59,7 +59,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" + echo "The kernel does not exist" exit } -- 2.23.0

1 0

[PATCH compass-ci 3/3] build-my-info: fix name mismatches
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index d0ee46e..fd75f51 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -15,11 +15,11 @@ option = { options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| - option[:email] = email + option[:my_email] = email end opts.on('-n name', 'my_name') do |name| - option[:name] = name + option[:my_name] = name end opts.on('-l lab', 'lab') do |lab| -- 2.23.0

1 0

[PATCH compass-ci 2/3] build-my-info: set default values
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 0b16884..d0ee46e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -7,7 +7,12 @@ require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' require 'optparse' -option = {} +option = { + my_name: `git config --global user.name`.chomp, + my_email: `git config --global user.email`.chomp, + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp +} + options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email -- 2.23.0

1 0

[PATCH compass-ci 1/3] build-my-info: fix white space style
by Wu Fengguang 09 Jan '21

09 Jan '21

Layout/SpaceAroundOperators: Surrounding space missing for operator `=`. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index cd824c6..0b16884 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -8,7 +8,7 @@ require_relative '../lib/build_my_info_client' require 'optparse' option = {} -options= OptionParser.new do |opts| +options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email end -- 2.23.0

1 0