- Compass-ci - mailweb.openeuler.org

[PATCH v1 compass-ci 1/3] extract-stats: close etcd client
by Cao Xueliang 04 Apr '21

04 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 11 +++++++---- src/extract-stats/stats_worker.cr | 5 +++-- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 4ae3e71..c35d3ec 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -6,10 +6,13 @@ require "./constants" require "./stats_worker" module ExtractStats + @@ec = EtcdClient.new + def self.consume_tasks channel = Channel(String).new - revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) - self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + queue = EXTRACT_STATS_QUEUE_PATH + revision = self.consume_by_list(queue, channel) + self.consume_by_watch(queue, revision, channel) end def self.consume_by_list(queue, channel) @@ -26,7 +29,7 @@ module ExtractStats def self.get_history_tasks(queue) tasks = [] of Etcd::Model::Kv - range = EtcdClient.new.range_prefix(queue) + range = @@ec.range_prefix(queue) revision = range.header.not_nil!.revision tasks += range.kvs @@ -45,7 +48,7 @@ module ExtractStats end def self.watch_queue(queue, revision, channel) - watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64 + 1, filters: [Etcd::Watch::Filter::NODELETE]) do |events| events.each do |event| puts event channel.send(event.kv.key) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 8910951..45b75a3 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,6 +1,5 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" @@ -30,6 +29,8 @@ class StatsWorker STDERR.puts e.message # incase of many error message when task-queue, ES does not work sleep(10) + ensure + @etcd.close end end @@ -71,7 +72,7 @@ class StatsWorker sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" - value = {"error_id" => sample_error_id} + value = {"job_id" => job_id, "error_id" => sample_error_id} @etcd.put(queue, value) msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: close etcd client
by Cao Xueliang 03 Apr '21

03 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 9 ++++++--- src/extract-stats/stats_worker.cr | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 4ae3e71..3346d12 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -6,10 +6,13 @@ require "./constants" require "./stats_worker" module ExtractStats + @@ec = EtcdClient.new + def self.consume_tasks channel = Channel(String).new - revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) - self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + queue = EXTRACT_STATS_QUEUE_PATH + revision = self.consume_by_list(queue, channel) + self.consume_by_watch(queue, revision, channel) end def self.consume_by_list(queue, channel) @@ -26,7 +29,7 @@ module ExtractStats def self.get_history_tasks(queue) tasks = [] of Etcd::Model::Kv - range = EtcdClient.new.range_prefix(queue) + range = @@ec.range_prefix(queue) revision = range.header.not_nil!.revision tasks += range.kvs diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 8910951..9dea267 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,6 +1,5 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" @@ -30,6 +29,8 @@ class StatsWorker STDERR.puts e.message # incase of many error message when task-queue, ES does not work sleep(10) + ensure + @etcd.close end end -- 2.23.0

1 0

review star
by Wu Fengguang 02 Apr '21

02 Apr '21

Cao Xueliang is our review star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 81 11 11 25 Cao Xueliang 59 12 19 12 Wei Jihui 24 13 28 5 Luan Shengde 22 7 6 3 Cui Yili 19 3 4 3 Wu Zhende 16 3 5 2 Yu Chuan 12 5 5 2 Xu Xijian 2 1 1 1 Wu Fengguang 25 0 0 1 Li Ping 17 1 1 0 Wang Chenglong 20 0 0 0 Liu Yinsi 6 0 0 0 Lu Weitao 1 0 0 0 Zhang Dewan 20 1 1 0 Wang Yong 6 0 0 0 Li Yuanchao 1 0 0 0 Bai Jing 9 4 4 0 Lin Jiaxin 18 5 4 0 Liu Shaofei

1 0

review star
by Wu Fengguang 02 Apr '21

02 Apr '21

Luan Shengde is our review star, thanks Luan Shengde! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 81 11 11 25 Cao Xueliang 59 12 19 12 Wei Jihui 24 13 28 5 Luan Shengde 22 7 6 3 Cui Yili 19 3 4 3 Wu Zhende 16 3 5 2 Yu Chuan 12 5 5 2 Xu Xijian 2 1 1 1 Wu Fengguang 25 0 0 1 Li Ping 17 1 1 0 Wang Chenglong 20 0 0 0 Liu Yinsi 6 0 0 0 Lu Weitao 1 0 0 0 Zhang Dewan 20 1 1 0 Wang Yong 6 0 0 0 Li Yuanchao 1 0 0 0 Bai Jing 9 4 4 0 Lin Jiaxin 18 5 4 0 Liu Shaofei

2 1

help star
by Wu Fengguang 02 Apr '21

02 Apr '21

Wu Zhende, Yu Chuan is our help star again, thanks Wu Zhende, Yu Chuan! baijing@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-04-01.md 14.3 Wu Zhende 14.2 Yu Chuan 14.0 Wei Jihui 11.3 Wu Fengguang 11.0 Liu Shaofei 10.7 Wang Yong 10.2 Zhang Dewan 9.4 Lin Jiaxin 7.5 Luan Shengde 6.8 Cao Xueliang 5.8 Liu Yinsi 5.4 Bai Jing 4.4 Lu Weitao 4.2 Du Kaitian 4.0 Li Ping 2.2 Cui Yili 1.8 Wang Chenglong 1.5 Li Yuanchao 1.4 Xu Xijian 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 02 Apr '21

02 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 19 Cao Xueliang 9 Liu Yinsi 8 Wu Zhende 6 Liu Shaofei 6 Lu Weitao 6 Wang Yong 4 Xu Xijian 4 Cui Yili 4 Luan Shengde 3 Wei Jihui 3 Yu Chuan 3 Li Yuanchao 3 Lin Jiaxin 3 Li Ping 2 Wang Chenglong 1 Bai Jing 1 Zhang Dewan 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale

1 0

[PATCH lkp-tests] test/ansible_test: modify ansible code to output the module name
by Wang Chenglong 02 Apr '21

02 Apr '21

[Why] modify ansible code to output the name of the module for tasks execution. [How] print("{'action':'%s'}" % task_result.__dict__['_task_fields']['action']) {'action':'package'} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/ansible_test b/tests/ansible_test index 56926bbed..1c19c2b0f 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -58,7 +58,18 @@ ansible_playbook_run() EOF [ $? -eq 0 ] && { echo "Playbook has been generated!" - } + } + [ $(command -v ansible-playbook) ] || { + echo "ansible-playbook: command not found" + exit 1 + } + + sed -i "536a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + + sed -i "550a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + ansible-playbook "$ansible_path"/site.yml if [ $? -eq 0 ]; then -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats/ansible_test: optimize the output of error_id
by Wang Chenglong 02 Apr '21

02 Apr '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 101 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 85 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..d7f084027 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,109 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Could not find .*):/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

2 1

[PATCH compass-ci] service/scheduler: modify the data saved in the testbox
by Wu Zhende 02 Apr '21

02 Apr '21

1. add name: for search in es 2. add type: vm/dc/physical 3. delete history content: no need this info testbox's history info can find in logging Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 19 ++++++++++++++++++- src/scheduler/elasticsearch_client.cr | 7 ++----- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 305d070..2e4fd30 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -160,18 +160,35 @@ class Sched state = "requesting" end + type = get_type(testbox) queues = JSON.parse(queues.to_json) hash = { "job_id" => job_id, "state" => state, "time" => get_time, "deadline" => deadline, - "queues" => queues + "queues" => queues, + "type" => type, + "name" => testbox } + @redis.update_wtmp(testbox.to_s, hash) @es.update_tbox(testbox.to_s, hash) end + def get_type(testbox) + return unless testbox + + if testbox.includes?("vm") + type = "vm" + elsif testbox.includes?("dc") + type = "dc" + else + type = "physical" + end + type + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 219cf3d..f32256b 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -117,17 +117,14 @@ class Elasticsearch::Client result = @client.get_source(query) raise result unless result.is_a?(JSON::Any) - history = result["history"].as_a? result = result.as_h else result = wtmp_hash end - history ||= [] of JSON::Any - history << JSON.parse(wtmp_hash.to_json) unless wtmp_hash["state"]?.to_s == "requesting" - history = JSON.parse(history.to_json) - + history = JSON::Any.new([] of JSON::Any) body = { "history" => history} + body.any_merge!(result) body.any_merge!(wtmp_hash) -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: add parameters annotation
by Wang Yong 02 Apr '21

02 Apr '21

it's for choosing parameters clearly Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 7a101b6a0..a91ae4a08 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,6 +3,7 @@ # - compat_os # - repo_name # - repo_addr +# Git repo jobs only use the first two parameters, SRPMs jobs use all . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -- 2.23.0

1 0