- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/job.rb: add define files when program is cci-depends
by Wei Jihui 17 Oct '20

17 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..9a08dc30 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -617,9 +617,9 @@ class Job program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| - # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg + # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files - if %w(makepkg makepkg-deps pack-deps cci-makepkg).include?(program) + if %w(makepkg makepkg-deps pack-deps cci-makepkg cci-depends).include?(program) program = programs_params_hash[program]["benchmark"] end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides to defaults
by Wei Jihui 17 Oct '20

17 Oct '20

[why] submit job.yaml testbox=vm-2p8g it should use vm-2p8g to load host file, not use defaults's testbox [slove] merge @overrides to @defaults, then use @defaults to load host file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..4e398a5a 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -382,7 +382,8 @@ class Job if defaults.is_a?(Hash) && !defaults.empty? @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) - load_hosts_config if check_load_hosts_config(defaults) + @defaults.merge!(@overrides) + load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: set flag true only when find hosts file
by Wei Jihui 17 Oct '20

17 Oct '20

it should set load_hosts_done is true only when find and load hosts file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..a5fe450f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -307,10 +307,10 @@ class Job hwconfig = load_yaml(hosts_file, nil) @job[source_file_symkey(hosts_file)] = nil @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig + @load_hosts_done = true end @job['arch'] ||= 'aarch64' - @load_hosts_done = true end def check_set_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: return the monitored logs
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] When other people use monitor to get logs, the logs only output, not return. It's hard to deal the logs. [How] Append result logs to a list. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 2a10b5c1..baf172d3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,6 +25,7 @@ class Monitor @exit_status_code = 0 @defaults = {} load_default + @result = [] end def load_default @@ -122,6 +123,7 @@ class Monitor ws.on :message do |event| data = JSON.parse(event.data) + @result << data output(data) connect(data, ws) -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix ameba warning
by Wu Zhende 17 Oct '20

17 Oct '20

parse_serial_logs.cr:46:5 [W] Lint/UselessAssign: Useless assignment to variable `host` > host = File.basename(msg["serial_path"].to_s) ^ parse_serial_logs.cr:95:5 [W] Lint/UselessAssign: Useless assignment to variable `job_id` > job_id = matched.named_captures["job_id"] ^ parse_serial_logs.cr:101:5 [W] Lint/UselessAssign: Useless assignment to variable `job` > job = Elasticsearch::Client.new.get_job_content(job_id) ^ Finished in 10.93 milliseconds 1 inspected, 3 failures. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index a000ca3..6c8a220 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -43,7 +43,7 @@ class SerialParser def host_in_msg(msg) return unless msg["serial_path"]? - host = File.basename(msg["serial_path"].to_s) + File.basename(msg["serial_path"].to_s) end def detect_start_or_end(msg, host, pattern_list) @@ -92,13 +92,13 @@ class SerialParser matched = msg["message"].to_s.match(/.*\/job_initrd_tmpfs\/(?<job_id>.*?)\//) return unless matched - job_id = matched.named_captures["job_id"] + matched.named_captures["job_id"] end def find_job(job_id) return unless job_id - job = Elasticsearch::Client.new.get_job_content(job_id) + Elasticsearch::Client.new.get_job_content(job_id) end def dump_cache(job, msg, host) -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: forward docker logs to master-fluentd
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 33 ++++++++++----------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 8e76441..ac73dfb 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -60,28 +60,19 @@ </store> <store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - num_threads 2 - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host 172.17.0.1 + port 24225 + </server> </store> - </match> <label @FLUENT_LOG> -- 2.23.0

1 0

[PATCH lkp-tests] skip build these container for multi-docker
by Xiao Shenwei 17 Oct '20

17 Oct '20

multi-docker do't need thses container, skip it Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- setup/simplify-ci | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f77..eb93c150 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -36,23 +36,22 @@ boot_ipxe() run_service() { + [[ $suite == "multi-docker" ]] && return + ( cd $CCI_SRC/container/dnsmasq || return - ./build - ./start + ./build && ./start boot_ipxe )& ( cd $CCI_SRC/container/qemu-efi || return - ./build - ./install + ./build && ./install )& ( cd $CCI_SRC/container/fluentd-base || return ./build cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start + ./build && ./start )& } -- 2.23.0

1 1

[PATCH compass-ci] container/master-fluentd: add a new fluentd container
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/Dockerfile | 6 +++ container/master-fluentd/build | 6 +++ container/master-fluentd/build-depends | 1 + container/master-fluentd/docker-fluentd.conf | 48 ++++++++++++++++++++ container/master-fluentd/start | 26 +++++++++++ 5 files changed, 87 insertions(+) create mode 100644 container/master-fluentd/Dockerfile create mode 100755 container/master-fluentd/build create mode 100755 container/master-fluentd/build-depends create mode 100644 container/master-fluentd/docker-fluentd.conf create mode 100755 container/master-fluentd/start diff --git a/container/master-fluentd/Dockerfile b/container/master-fluentd/Dockerfile new file mode 100644 index 0000000..192ece3 --- /dev/null +++ b/container/master-fluentd/Dockerfile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fluentd-base:alpine + +COPY --chown=1090:1090 docker-fluentd.conf /fluentd/etc/docker-fluentd.conf diff --git a/container/master-fluentd/build b/container/master-fluentd/build new file mode 100755 index 0000000..7d0291d --- /dev/null +++ b/container/master-fluentd/build @@ -0,0 +1,6 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +system 'docker build -t master-fluentd:alpine .' diff --git a/container/master-fluentd/build-depends b/container/master-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/master-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf new file mode 100644 index 0000000..ef97d5c --- /dev/null +++ b/container/master-fluentd/docker-fluentd.conf @@ -0,0 +1,48 @@ +<system> + workers 10 +</system> + +<source> + @type forward + bind 0.0.0.0 +</source> + +<match **> + @type copy + + <store> + @type elasticsearch + host 172.17.0.1 + port 9202 + suppress_type_name true + flush_interval 1s + index_name ${tag} + ssl_verify false + log_es_400_reason true + with_transporter_log true + </store> + + <store> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> + </store> +</match> + +<label @FLUENT_LOG> + <match fluent.*> + @type stdout + <format> + @type stdout + output_type single_value + message_key message + add_newline true + </format> + </match> +</label> diff --git a/container/master-fluentd/start b/container/master-fluentd/start new file mode 100755 index 0000000..9e02980 --- /dev/null +++ b/container/master-fluentd/start @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "master-fluentd" + +cmd = %w[ + docker run + --restart=always + --name master-fluentd + -v /etc/localtime:/etc/localtime:ro + -d + -u 1090:1090 + -p 24225:24224/tcp + -p 24225:24224/udp + -e FLUENTD_CONF=docker-fluentd.conf + --log-driver json-file + --log-opt max-size=1g + master-fluentd:alpine +] +cmd += ['sh', '-c', 'umask 002 & fluentd -c /fluentd/etc/docker-fluentd.conf'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 32 +++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..285c393 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +def generate_conserver(lab) + return unless File.exist?('conserver-head.cf') + + FileUtils.cp('conserver-head.cf', 'conserver.cf') + + host_dir = "lab-#{lab}/hosts/" + return unless Dir.exist?(host_dir) + + Dir.each_child(host_dir) do |host| + ipmi_ip = YAML.load_file("#{host_dir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + console = <<~HEREDOC + console #{host} { + exec /usr/local/bin/ipmi.sh #{ipmi_ip}; + } + HEREDOC + File.open('conserver.cf', 'a') { |f| f << console } + end +end + +generate_conserver(lab) -- 2.23.0

1 0

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 30 +++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..1c120b1 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +FileUtils.cp('conserver-head.cf', 'conserver.cf') + +def generate_conserver(lab) + hostdir = "lab-#{lab}/hosts/" + return unless Dir.exist?(hostdir) + + puts hostdir + Dir.each_child(hostdir) do |host| + ipmi_ip = YAML.load_file("#{hostdir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + File.open('conserver.cf', 'a') do |f| + f << "console #{host} {\n" + f << " exec /usr/local/bin/ipmi.sh #{ipmi_ip}\n" + f << "}\n" + end + end +end + +generate_conserver(lab) -- 2.23.0

1 0