March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/stats_wrapper.rb: warn if stats-script doesn't exists
by Lu Weitao 31 Mar '21

31 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index faa81df..d7ef9ce 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -29,6 +29,11 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" + unless File.exist?("#{PROGRAM_DIR}/#{@program}") + warn "#{PROGRAM_DIR}/#{@program} doesn't exist" + return + end + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile -- 2.23.0

1 0

Re: [PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Li Ping 31 Mar '21

31 Mar '21

rename var from vm_name to docker_name ---------^ Thanks, Ping Reply-To: In-Reply-To: <20210330074124.1780870-1-weijihuiall(a)163.com> On Tue, Mar 30, 2021 at 03:41:24PM +0800, Wei Jihui wrote: >vm_name --> docker_name > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > container/ssh-r/start | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > >diff --git a/container/ssh-r/start b/container/ssh-r/start >index 68a63f2..47a29eb 100755 >--- a/container/ssh-r/start >+++ b/container/ssh-r/start >@@ -4,14 +4,14 @@ > > . $CCI_SRC/container/defconfig.sh > >-vm_name=ssh_r >+docker_name=ssh_r > >-docker_rm ${vm_name} >+docker_rm ${docker_name} > > cmd=( > docker run > --restart=always >- --name ${vm_name} >+ --name ${docker_name} > -e GATEWAY_PORTS=true > -e TCP_FORWARDING=true > -d >-- >2.23.0 >

1 0

[PATCH compass-ci] providers/docker: replace URI.open to %x(curl url)
by Cao Xueliang 31 Mar '21

31 Mar '21

curl can suspend when the response need more time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/docker.rb | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 51017f3..db0ccee 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -39,15 +39,10 @@ def del_host2queues(hostname) end def parse_response(url) - response = nil - URI.open(url) do |http| - response = http.read - end + response = %x(curl #{url}) hash = response.is_a?(String) ? JSON.parse(response) : nil if hash.nil? || !hash.key?('job') - puts '..........' - puts 'no job now' - puts '..........' + puts response return nil end return hash -- 2.23.0

1 0

[PATCH compass-ci] docker/run.sh: do not forward logs
by Wu Zhende 31 Mar '21

31 Mar '21

After the configuration, docker logs are dumped tp fluentd. The temporary docker also generates an index in logging-es. This is unnecessary and wastes es resources. Save temporary docker logs to the local host. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 80950dd..516ebbc 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -36,6 +36,8 @@ cmd=( -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -v ${busybox_path}:/usr/local/bin/busybox + --log-driver json-file + --log-opt max-size=10m --oom-score-adj="-1000" ${docker_image} /root/sbin/entrypoint.sh -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test:
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 104 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 87 insertions(+), 17 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..0ede72c8f 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,110 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end -while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + #when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end - ansible_failed_json = JSON.parse @ansible_failed_info +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case ansible_failed_json['failures'] + when /.*/ + p '1' + else + output_error ansible_failed_json['failures'] + end + end +end + +while (line = STDIN.gets) + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + puts "module_name: #{@module_name['action']}" + when /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 1

[PATCH v2 compass-ci 2/2] src/monitor: detect crash and send msg to mq queue.
by Li Ping 30 Mar '21

30 Mar '21

linux/kernel/panic.c: pr_emerg("Kernel panic - not syncing: %s\n", buf); When kernel panic happens, it will print the panic call stack, then the ending msg like: [ 14.405869] Kernel panic - not syncing: Fatal exception in interrupt some bug not cause kernel panic happens but also mean that job fails like: mount.nfs: Connection timed out No space left on device The above message would be detected, then send msg to mq queue to close job and reboot machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 47 +++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 451312c..ff3b84a 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,16 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + "Kernel panic - not syncing: ", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -51,7 +59,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, pattern_list) + def detect_patterns(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,17 +68,50 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, signal) + boundary_signal = detect_patterns(msg, signal) return unless boundary_signal @host2head.delete(host) @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_patterns(msg, CRASH_PATTERNS) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] src/lib: add on_job_crash
by Li Ping 30 Mar '21

30 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index adcb965..0e6cb82 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -189,6 +191,16 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + close_job(event_job_id, "crash") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -296,6 +308,15 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + reboot_machine(testbox, machine) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) @@ -308,6 +329,10 @@ class Lifecycle deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) return if Time.local < deadline + reboot_machine(testbox, machine) + end + + def reboot_machine(testbox, machine) mq_queue = get_machine_reboot_queue(testbox) machine.as_h.delete("history") machine.as_h["testbox"] = JSON::Any.new(testbox) -- 2.23.0

1 0

[PATCH compass-ci] etcd/Dockerfile: change source address
by Wang Yong 30 Mar '21

30 Mar '21

accelerate download Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/etcd/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index 3654fc4..b3957e8 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ARG VERSION RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && apk add ca-certificates && \ - wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-arm64.ta… && \ tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* -- 2.23.0

1 0

[PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] container/taskqueue: compiling code every time
by Wu Zhende 30 Mar '21

30 Mar '21

Delete the complied taskqueue file. Otherwise, the code will be updated and the taskqueue file will not be recomplied. Uses old code all the time. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/taskqueue/build | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/container/taskqueue/build b/container/taskqueue/build index 4f69f85..518a27e 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -12,10 +12,8 @@ service_port=${TASKQUEUE_PORT:-3060} docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" -if [[ ! -f "taskqueue" ]]; then - $CCI_SRC/container/compile . -else - echo -e "\n\033[34m- Info: using \033[31mexists\033[34m taskqueue -\033[0m\n" -fi +$CCI_SRC/container/compile . docker build -t taskqueue-alpine:v0.001-${service_port} . + +rm taskqueue -- 2.23.0

1 0