- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/2] scheduler/elasticsearch_client.cr: add two account functions
by Wu Zhende 24 Nov '20

24 Nov '20

add two functions about account 1. get_account 2. update_account Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..7c92204 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,24 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + query = {:index => "accounts", :type => "_doc", :id => my_email} + response = JSON.parse({"_id" => my_email, "found" => false}.to_json) + return response unless @client.exists(query) + + @client.get_source(query) + end + + def update_account(account_content : JSON::Any, my_email : String) + return @client.update( + { + :index => "accounts", :type => "_doc", + :id => my_email, + :body => {:doc => account_content} + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

2 1

[PATCH compass-ci] srv-http/start: add os images data download directory
by Cui Yili 24 Nov '20

24 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/job.cr: check account info when submit job
by Wu Zhende 24 Nov '20

24 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid. Check whether the information is the same as that stored in the ES database. Also need update pub_key. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 51 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..785daea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,11 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") + @account_info = Hash(String, JSON::Any).new check_required_keys() + check_account_info() set_defaults() end @@ -152,7 +156,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() - set_sshr_port() + set_sshr_info() set_queue() end @@ -183,12 +187,34 @@ class Job end end - private def set_sshr_port + private def set_sshr_info return unless self["sshd"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + + return if @account_info["found"]? == false + + update_sshd + end + + private def update_sshd + sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() + pub_key = sshd["pub_key"]?.to_s + update_account_my_pub_key(pub_key) + + sshd["pub_key"] = @account_info["my_ssh_pubkey"] + @hash["sshd"] = JSON.parse(sshd.to_json) + end + + private def update_account_my_pub_key(pub_key) + my_ssh_pubkey = @account_info["my_ssh_pubkey"].as_a + return if pub_key.empty? || my_ssh_pubkey.includes?(pub_key) + + my_ssh_pubkey << JSON::Any.new(pub_key) + @account_info["my_ssh_pubkey"] = JSON.parse(my_ssh_pubkey.to_json) + @es.update_account(JSON.parse(@account_info.to_json), self["my_email"].to_s) end private def set_os_dir @@ -271,6 +297,9 @@ class Job id suite testbox + my_email + my_name + my_uuid ] private def check_required_keys @@ -281,6 +310,20 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + raise account_info unless account_info.is_a?(JSON::Any) + + @account_info = account_info.as_h + + raise error_msg if @account_info["found"]? == false + raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +345,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +486,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] stats/mysql-server: remove redundant whitespace
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/mysql-server | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..776f3f2e 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -16,9 +16,9 @@ while (line = STDIN.gets) # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 when /\[\s*\d+\%\]/ - results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') - results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') - results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') + results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') + results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') + results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') end end -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..e2ab4b8e --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

2 1

[PATCH compass-ci] optimize document for apply account
by Luan Shengde 24 Nov '20

24 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index d718d0d..b8dd4bb 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -36,20 +36,12 @@ for external user email subject: apply account email body: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… attachment: ssh pubkey file - it will return email with my_email, my_name, my_uuid if successfully applied account -2. prepare to submit job - execute on user's local host - - config default yaml file: - ~/.config/compass-ci/default/account.yaml - - prepare job.yaml - - submit job - -3. apply account email format - example: + - email example: To: compass-ci(a)qq.com Subject: apply account @@ -58,5 +50,20 @@ for external user # We'll validate whether the URL contains your email. # for example, my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - # attach your ssh pub key to the email as attachment + +2. successfully applied account email + it will returns email with following info: + - my_email + - my_name + - my_uuid + - sched_host + - schec_port + +3. environment configuration + you will get email after successfully applied account + follow steps in the email to finish the following configuration + - setup default yaml + ~/.config/compass-ci/default/account.yaml + - download lkp-tests and dependencies + - prepare to submit jobs -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] scheduler/elasticsearch_client.cr: add two account function
by Wu Zhende 24 Nov '20

24 Nov '20

add two function about account Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..efe304c 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,28 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = "" + end + + response = JSON.parse({"_id" => my_email, "found" => false}.to_json) if response.is_a?(String) + + return response + end + + def update_account(account_content : JSON::Any, my_email : String) + return @client.update( + { + :index => "accounts", :type => "_doc", + :id => my_email, + :body => {:doc => account_content} + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

3 3

[PATCH lkp-tests] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..ce3903ef --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

1 0

[PATCH v6 compass-ci 1/2] result-webdav/start: add share dir to upload file
by Cui Yili 24 Nov '20

24 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

2 1

[PATCH compass-ci] about multi-device test
by Ren Wen 24 Nov '20

24 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- doc/multi-device-test.md | 108 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 doc/multi-device-test.md diff --git a/doc/multi-device-test.md b/doc/multi-device-test.md new file mode 100644 index 0000000..226287f --- /dev/null +++ b/doc/multi-device-test.md @@ -0,0 +1,108 @@ +# Multi-device Test # + +��, ��. + +## When using multi-device ## + + * �� + * �� + +## How to run a multi-device test ## + +��job.yaml��, ��**cluster**��, �� +��, ��: *lab-z9/cluster/*. +��job��job, �� +��. + +��? +��job.yaml��: + +``` yaml +if role <role01> + <test_script01> + <test_script02> + +if role <role02> + <test_script03> + <test_script04> +... +``` + +��, ��. �� +job��, ��. + +��: ��*role01*, �� +*test_script01*, *test_script02*, ��. + +### how to find test script ### + +��lkp��**daemon**, **tests**��. +��**daemon**��*test_script01*, ��`start_daemon`�� +��. ��**tests**��*test_script01*, ��`run_test`�� +��. ��, ��. + +### about `start_daemon` and `run_test` ### + +��2��**$LKP_SRC/lib/job.sh**. +��, ��. + +#### the processes of `start_damon` #### + + 1. ��**daemon**��. ��, ��: + `started`, ��: `start`; ��, ��: + `failed`, ��: `abort`, ��. ��redis�� + key: `HGETALL sched/cluster_state`. + 2. ��: `write_state`, ��: + * `node_roles`: �� + * `ip`: ��`hostname -I`�� + * `direct_macs`: ��**macs**�� + * `direct_ips`: ��**ip0**��ip, �� + ��**macs**�� + ��, ��redis��key: + **sched/cluster_state**. + 3. ��**direct_ips**��**direct_macs**��. + 4. ��: `wait_ready`, ��: `ready`; �� + ��, ��. �� + ��, ��`ready`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 5. ��: `roles_ip`, ��. + ��: `ip=ip01`��, ��: `export ip=ip01`. + 6. ��: `wait_finish`, ��: `finish`; �� + ��, ��`finish`. �� + ��`finish`, ��`finish`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 7. ��. + +#### the processes of `run_test` #### + +��1,2,3,4��2,3,4,5. + + 1. ��: `write_state`, ��: + * `node_roles`: �� + * `ip`: ��`hostname -I`�� + * `direct_macs`: ��**macs**�� + * `direct_ips`: ��**ip0**��ip, �� + ��**macs**�� + ��, ��redis��key: + **sched/cluster_state**. + 2. ��**direct_ips**��**direct_macs**��. + 3. ��: `wait_ready`, ��: `ready`; �� + ��, ��. �� + ��, ��`ready`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 4. ��: `roles_ip`, ��. + ��: `ip=ip01`��, ��: `export ip=ip01` + 5. ��**tests**��. ��, ��: + `finished`, ��: `finish`; ��, ��: `failed`, �� + ��: `abort`, ��. + 6. ��. + +#### other #### + +`start_daemon`��`run_test`��`abort`��: ��100�� +��, ��`ready`��`finish`, �� +`retry`, ��, ��. + +��, ��(��)��, �� +��`ready`(`finish`)��`abort`. ��100��, �� +��`abort`��. -- 2.23.0

2 4