- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/kibana: add logtrail plugin
by Wu Zhende 08 Dec '20

08 Dec '20

LogTrail is a plugin for Kibana to view, analyze, search, and tail log events from multiple hosts in realtime with DevOps friendly interface inspired by Papertrail. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/Dockerfile | 10 +++++++++ container/kibana/logtrail.json | 41 ++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 container/kibana/logtrail.json diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 970eb5a..60b889d 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -10,3 +10,13 @@ FROM ${BASE_IMAGE} MAINTAINER Wu Zhende <wuzhende666(a)163.com> RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml + +USER root +RUN yum -y install wget \ + && wget https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-7.… -O /logtrail-7.6.2-0.1.31.zip + +USER 1090 +RUN ./bin/kibana-plugin install file:///logtrail-7.6.2-0.1.31.zip \ + && rm -rf /tmp/logtrail-7.6.2-0.1.31.zip + +COPY --chown=1090:1090 logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json new file mode 100644 index 0000000..304a4b3 --- /dev/null +++ b/container/kibana/logtrail.json @@ -0,0 +1,41 @@ +{ + "version": 2, + "index_patterns": [{ + "es": { + "default_index": "scheduler-3016", + "allow_url_parameter": false, + "timezone": "CST" + }, + "tail_interval_in_seconds": 10, + "es_index_time_offset_in_seconds": 0, + "display_timezone": "CST", + "display_timestamp_format": "YYYY MMM DD HH:mm:ss", + "max_buckets": 500, + "nested_objects": false, + "default_time_range_in_days": 0, + "max_hosts": 100, + "max_events_to_keep_in_viewer": 5000, + "default_search": "", + "fields": { + "mapping": { + "timestamp": "time", + "display_timestamp" : "time", + "hostname": "container_name", + "program": "tags", + "message": "log" + }, + "message_format": "{{{log}}}", + "keyword_suffix": "keyword" + }, + "color_mapping": { + "field": "level", + "mapping": { + "ERROR": "#FF0000", + "WARN": "#FFEF96", + "DEBUG": "#B5E7A0", + "TRACE": "#CFE0E8", + "INFO": "#339999" + } + } + }] +} -- 2.23.0

1 0

[PATCH lkp-tests] modify install-dependencies.sh permission
by Hu Xuejiao 08 Dec '20

08 Dec '20

[why] Error: Running Homebrew as root is extremely dangerous and no longer supported. As Homebrew does not drop privileges on installation you would be giving all build scripts full access to your system. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 907dba9e..0b2d09a2 100644 --- a/Makefile +++ b/Makefile @@ -15,7 +15,7 @@ install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp bash sbin/set-env.sh - sudo bash sbin/install-dependencies.sh + bash sbin/install-dependencies.sh .PHONY: doc doc: -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/json_logger: fix time format
by Wu Zhende 08 Dec '20

08 Dec '20

[Why] Time requires a specific format to be resolved by ES to date type [Reference] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-dat… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index d136bdd..9596243 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -15,6 +15,7 @@ class JSONLogger < Logger Logger::Formatter.new do | severity, datetime, progname, msg, io| get_env_info(@env.as(HTTP::Server::Context)) if @env level_num = severity.to_i32 + datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S") logger_hash = JSON.parse(%({"level_num": #{level_num}, "level": "#{severity}", "time": "#{datetime}" -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/master-fluentd: fix time format
by Wu Zhende 08 Dec '20

08 Dec '20

[Why] Time requires a specific format to be resolved by ES to date type [Reference] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-dat… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 5de7901..b3daeb4 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -11,7 +11,7 @@ @type record_transformer enable_ruby <record> - time ${time.strftime('%Y-%m-%d %H:%M:%S%z')} + time ${time.strftime('%Y-%m-%dT%H:%M:%S')} </record> </filter> -- 2.23.0

1 0

[PATCH compass-ci] src/lib: fix job.cr
by Zhang Yu 08 Dec '20

08 Dec '20

1. fix ameba warning job.cr:428:32 [C] Layout/TrailingWhitespace: Trailing whitespace detected > temp_initrds = [] of String ^ 2. fix no need () for no parameter in Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 196516f..7746889 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -333,7 +333,7 @@ class Job @hash.delete("my_uuid") end - private def get_initialized_keys() + private def get_initialized_keys initialized_keys = [] of String REQUIRED_KEYS.each do |key| @@ -425,7 +425,7 @@ class Job end private def get_initrds - temp_initrds = [] of String + temp_initrds = [] of String if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib: add delete function in job.cr
by Zhang Yu 08 Dec '20

08 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4345e0..70eab23 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -136,6 +136,15 @@ class Job update(json.as_h) end + def delete(key : String) + initialized_keys = get_initialized_keys + if initialized_keys.includes?(key) + raise "Should not delete #{key}" + else + @hash.delete(key) + end + end + private def set_defaults append_init_field() set_user_lkp_src() @@ -324,7 +333,7 @@ class Job @hash.delete("my_uuid") end - private def initialized? + private def get_initialized_keys() initialized_keys = [] of String REQUIRED_KEYS.each do |key| @@ -350,7 +359,10 @@ class Job "kernel_uri", "kernel_params", "linux_vmlinuz_path"] + end + private def initialized? + initialized_keys = get_initialized_keys initialized_keys.each do |key| return false unless @hash.has_key?(key) end -- 2.23.0

2 2

[PATCH lkp-tests] setup/netdata: use netdata monitor system resources
by Xiao Shenwei 08 Dec '20

08 Dec '20

usage: suite: multi-qemu simplify-ci: netdata: netdata_master_host: netdata_master_port: netdata_uuid: Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- setup/netdata | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 setup/netdata diff --git a/setup/netdata b/setup/netdata new file mode 100755 index 00000000..b318dc7c --- /dev/null +++ b/setup/netdata @@ -0,0 +1,45 @@ +#!/bin/sh +# monitoring system resources +# depend on $LKP_SRC/setup/simplify-ci + +check_env() +{ + [ -d "/c/compass-ci" ] || { + echo "please setting the compass-ci environment" + return 1 + } +} + +check_variables() +{ + [[ -n "$netdata_master_host" && -n "$netdata_master_port" && -n "$netdata_uuid" ]] || { + echo "please check the following variables:" + echo "1. netdata_master_host" + echo "2. netdata_master_port" + echo "3. netdata_uuid" + return 1 + } +} + +modify_config_file() +{ + cd /c/compass-ci/container/netdata-slave/config + sed -i "s%destination =.*%destination = $netdata_master_host:$netdata_master_port%g" stream.conf + sed -i "s%api key =.*%api key = $netdata_uuid%g" stream.conf +} + +run_container() +{ + cd /c/compass-ci/container/netdata-slave + ./build && ./run +} + +main() +{ + check_env || return + check_variables || return + modify_config_file + run_container +} + +main -- 2.23.0

2 1

[PATCH compass-ci] container/mail-robot: my_uuid is per-lab account
by Wu Fengguang 08 Dec '20

08 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 3 +++ container/mail-robot/start | 1 + 2 files changed, 4 insertions(+) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 0c5f8d4..a88626c 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -24,6 +24,9 @@ def build_apply_account_email(my_info) cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} + EOF + mkdir -p ~/.config/compass-ci/include/lab + cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF my_uuid: #{my_info['my_uuid']} EOF diff --git a/container/mail-robot/start b/container/mail-robot/start index f8078b6..ad17c7a 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -16,6 +16,7 @@ names = Set.new %w[ SCHED_HTTPS_PORT ES_PORT MAILDIR + lab ] defaults = relevant_defaults(names) -- 2.23.0

2 3

[PATCH v2 lkp-tests] spec/submit_spec: submit spec without my info
by Wei Jihui 08 Dec '20

08 Dec '20

delete my-info like my_uuid, my_email, my_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 8310d629..77b64a7a 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -3,7 +3,7 @@ require "#{LKP_SRC}/lib/yaml" def stable_yaml_file(yaml_file) job = load_yaml(yaml_file) - delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin) + delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_uuid my_email my_name) job.delete_if do |key, _| delete_job_key.include?(key) || key.to_s.start_with?('#! ') || -- 2.23.0

1 0

[PATCH compass-ci 2/2] send-internet-mail: update stored mails path
by Luan Shengde 08 Dec '20

08 Dec '20

use new mailbox for mails than has been sent. it will enable read separately with mutt [why] the old path will not support to read separately for mutt with -f [how] use a new mailbox to store the mails that have been sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 851904c..8fdaa0d 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -23,7 +23,7 @@ Mail.defaults { delivery_method :smtp, smtp } def store_email(mail) time_now = Time.new.strftime('%Y%m%d%H%M%S') file_name = [mail.to[0], time_now].join('_') - file_full_name = File.join(ENV['MAILDIR'], 'sent', file_name) + file_full_name = File.join(ENV['SENT_MAILDIR'], 'new', file_name) File.open(file_full_name, 'w') do |f| f.puts mail end -- 2.23.0

1 0