- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] code-spec: grep-friendly fixed substring
by Wu Fengguang 22 Oct '20

22 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index 37c86b6..6f2dddd 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -19,6 +19,13 @@ code注意事项 - 如果注释仅仅是代码的简单重复, 不提供额外信息量, 不要写. +- puts/printf字符串应当容易搜索(grep)。特别注意包含足够长的固定内容substring. +例如： + die "test yaml$${test_yaml}$ not exist in ${LKP_SRC}/jobs !!!" +==> + die "cannot find test yaml ($test_yaml) in $LKP_SRC/jobs" + ^^^^^^^^^^^^^^^^^^^^^ grep-friendly fixed substring + 重构友好编码 ============ -- 2.23.0

1 0

[PATCH v2 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 22 Oct '20

22 Oct '20

[why] Provide a login port for users by email, determine whether to send the ssh_port message based on the "pub_key". Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index cd57be4f..1ebc9d00 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,16 +1,16 @@ #!/bin/sh # pub_key # my_email -# my_uuid -# ssh_port_len -# ssh_port_base +# sshr_port +# sshr_port_len +# sshr_port_base . $LKP_SRC/lib/http.sh sshr_ip=$LKP_SERVER -sshr_port=5050 -[ -n "$ssh_port_len" ] || ssh_port_len=2000 -[ -n "$ssh_port_base" ] || ssh_port_base=50000 +[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=50000 run_ssh() { @@ -23,6 +23,7 @@ run_ssh() data_success() { + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $my_email Subject: [NOTIFY Compass-ci] $testbox ready to use @@ -43,7 +44,7 @@ Dear $my_username: $os $os_version $os_arch Regards -Compass-ci +Compass-Ci " } @@ -54,10 +55,10 @@ Subject: [NOTIFY Compass-ci] Applying $testbox failed Dear $my_username: Sorry to inform you that your application failed. - You may need to wait for a while and then try again. + You may need to wait a moment, or check whether your pub_key exists. Regards -Compass-ci +Compass-Ci " } @@ -73,11 +74,11 @@ compose_email() set_port() { - [ -n "$my_uuid" ] || return + [ -n "$pub_key" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%"$ssh_port_len"+"$ssh_port_base")) + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return @@ -87,7 +88,7 @@ set_port() run_email() { - [ -n "$my_email" ] && [ -n "$my_uuid" ] || return 0 + [ -n "$my_email" ] || return 0 my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" -- 2.23.0

2 2

[PATCH v2 lkp-tests] sbin/submit: -m automatically stop monitoring
by Wu Zhende 22 Oct '20

22 Oct '20

[Why] When use "submit -m job.yaml" to monitor job state, it can't auto stop. Need to enter "Ctrl + c" manually. It's not convenient. So add the function of automatically stopping when "job_state=extract_finished" is received. [Output] connect to ws://localhost:11310/filter {"job_id"=>"z9.105511", "message"=>"", "job_state"=>"submit"} {"job_id"=>"z9.105511", "job_state"=>"boot"} {"job_id"=>"z9.105511", "job_state"=>"download"} {"time"=>"2020-10-22 11:14:55", "mac"=>"xx", "ip"=>"xx", "job_id"=>"z9.105511", "state"=>"running", "testbox"=>"vm-2p8g-3871595"} {"job_state"=>"running", "job_id"=>"z9.105511"} {"job_state"=>"post_run", "job_id"=>"z9.105511"} {"start_time"=>"2020-10-22 03:14:56", "end_time"=>"2020-10-22 03:15:26", "loadavg"=>"1.26 0.38 0.13 2/87 1268", "job_id"=>"z9.105511"} {"job_state"=>"finished", "job_id"=>"z9.105511"} {"job_id"=>"z9.105511", "job_state"=>"complete"} {"time"=>"2020-10-22 11:15:34", "mac"=>"xx", "ip"=>"xx", "job_id"=>"z9.105511", "state"=>"rebooting", "testbox"=>"vm-2p8g-3871595"} {"job_id"=>"z9.105511", "job_state"=>"extract_finished"} connection closed: Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/monitor | 6 ++++++ sbin/submit | 4 ++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/sbin/monitor b/sbin/monitor index c19a6056..def47bf6 100755 --- a/sbin/monitor +++ b/sbin/monitor @@ -14,6 +14,7 @@ opt_action = {'output' => true} opt_job = {} opt_monitor_url = nil opt_timeout = 0 +opt_stop_query = {} options = OptionParser.new do |opts| opts.banner = 'Usage: monitor [options] <filter>' @@ -49,6 +50,10 @@ options = OptionParser.new do |opts| opts.on('-j', '--job json', 'set job') do |job| opt_job.merge! YAML.load job end + + opts.on('-s', '--stop-query json', 'add query to stop monitor') do |stop_query| + opt_stop_query.merge! YAML.load stop_query + end end options.parse!(ARGV) @@ -75,5 +80,6 @@ monitor.monitor_url = opt_monitor_url if opt_monitor_url monitor.overrides = opt_set_key_value monitor.action = opt_action monitor.job = opt_job +monitor.stop_query = opt_stop_query monitor.run(opt_timeout) diff --git a/sbin/submit b/sbin/submit index 8b46db32..c49e1e9a 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,7 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} -actions = ['output'] +actions = ['output', 'stop'] options = OptionParser.new do |opts| opts.banner = 'Usage: submit [options] jobs...' @@ -157,6 +157,6 @@ end if opt_monitor opt_monitor_query.merge!({'job_id' => job_ids[0]}) - cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_json_list[0]}' -a #{actions.join(',')}" + cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_json_list[0]}' -a #{actions.join(',')} -s 'job_state: extract_finished'" exec cmd end -- 2.23.0

1 0

[PATCH lkp-tests] tests: add "arch" field with physical machine
by Zhang Yu 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index 46f43f93..d12ebc7e 100755 --- a/tests/host-info +++ b/tests/host-info @@ -10,9 +10,10 @@ get_node() echo "memory: $memory" } -get_model_name() +get_model_and_arch() { echo "model_name: $(lscpu |grep "Model name" | awk '{print $3}')" + echo "arch: $(lscpu |grep "Architecture" | awk '{print $2}')" } get_nr_disk() @@ -105,7 +106,7 @@ main() get_nr_disk get_partitions get_mac - get_model_name + get_model_and_arch get_ipmi_ip } -- 2.23.0

2 2

[PATCH lkp-tests] sbin/submit: -m automatically stop monitoring
by Wu Zhende 22 Oct '20

22 Oct '20

[Why] When use "submit -m job.yaml" to monitor job state, it can't auto stop. Need to enter "Ctrl + c" manually. It's not convenient. So add the function of automatically stopping when "job_state=extract_finished" is received. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/monitor | 6 ++++++ sbin/submit | 4 ++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/sbin/monitor b/sbin/monitor index c19a6056..def47bf6 100755 --- a/sbin/monitor +++ b/sbin/monitor @@ -14,6 +14,7 @@ opt_action = {'output' => true} opt_job = {} opt_monitor_url = nil opt_timeout = 0 +opt_stop_query = {} options = OptionParser.new do |opts| opts.banner = 'Usage: monitor [options] <filter>' @@ -49,6 +50,10 @@ options = OptionParser.new do |opts| opts.on('-j', '--job json', 'set job') do |job| opt_job.merge! YAML.load job end + + opts.on('-s', '--stop-query json', 'add query to stop monitor') do |stop_query| + opt_stop_query.merge! YAML.load stop_query + end end options.parse!(ARGV) @@ -75,5 +80,6 @@ monitor.monitor_url = opt_monitor_url if opt_monitor_url monitor.overrides = opt_set_key_value monitor.action = opt_action monitor.job = opt_job +monitor.stop_query = opt_stop_query monitor.run(opt_timeout) diff --git a/sbin/submit b/sbin/submit index 8b46db32..c49e1e9a 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,7 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} -actions = ['output'] +actions = ['output', 'stop'] options = OptionParser.new do |opts| opts.banner = 'Usage: submit [options] jobs...' @@ -157,6 +157,6 @@ end if opt_monitor opt_monitor_query.merge!({'job_id' => job_ids[0]}) - cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_json_list[0]}' -a #{actions.join(',')}" + cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_json_list[0]}' -a #{actions.join(',')} -s 'job_state: extract_finished'" exec cmd end -- 2.23.0

2 1

[PATCH lkp-tests] hackbench-20.yaml: fix "datesize" to "datasize"
by Zhang Yale 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- jobs/hackbench-20.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/hackbench-20.yaml b/jobs/hackbench-20.yaml index b69e8c77..12943c04 100644 --- a/jobs/hackbench-20.yaml +++ b/jobs/hackbench-20.yaml @@ -11,7 +11,7 @@ nr_threads: - 10 - 20 iterations: 10 -datesize: 1000 +datasize: 1000 runtime: 300 hackbench: -- 2.23.0

1 0

[PATCH lab-z9] hosts: add hostfile for a105-a107
by Zhang Yu 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a105 | 24 ++++++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a106 | 24 ++++++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a107 | 24 ++++++++++++++++++++++++ 3 files changed, 72 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a105 create mode 100644 hosts/taishan200-2280-2s48p-256g--a106 create mode 100644 hosts/taishan200-2280-2s48p-256g--a107 diff --git a/hosts/taishan200-2280-2s48p-256g--a105 b/hosts/taishan200-2280-2s48p-256g--a105 new file mode 100644 index 0000000..3c28d2d --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a105 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b891945d + - /dev/disk/by-id/scsi-350000399b8919135 + - /dev/disk/by-id/scsi-350000399b8919699 + - /dev/disk/by-id/scsi-350000399b89190d5 +mac_addr: + - 44:67:47:d7:76:c4 + - 44:67:47:d7:76:c5 + - 44:67:47:d7:76:c6 + - 44:67:47:d7:76:c7 + - 44:67:47:d7:76:c8 + - 44:67:47:d7:76:c9 + - 44:67:47:d7:76:ca + - 44:67:47:d7:76:cb + - cc:05:77:ee:b2:8f + - cc:05:77:ee:b2:90 + - cc:05:77:ee:b2:91 + - cc:05:77:ee:b2:92 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.10 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 new file mode 100644 index 0000000..a7b44a0 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b883bf99 + - /dev/disk/by-id/scsi-350000399b883ba89 + - /dev/disk/by-id/scsi-350000399b883b94d + - /dev/disk/by-id/scsi-350000399b883b981 +mac_addr: + - 44:67:47:86:1c:30 + - 44:67:47:86:1c:31 + - 44:67:47:86:1c:32 + - 44:67:47:86:1c:33 + - 44:67:47:86:1c:34 + - 44:67:47:86:1c:35 + - 44:67:47:86:1c:36 + - 44:67:47:86:1c:37 + - cc:05:77:ee:b7:df + - cc:05:77:ee:b7:e0 + - cc:05:77:ee:b7:e1 + - cc:05:77:ee:b7:e2 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.11 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 new file mode 100644 index 0000000..9d20e57 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8917c39 + - /dev/disk/by-id/scsi-350000399b8918005 + - /dev/disk/by-id/scsi-350000399b8900a4d + - /dev/disk/by-id/scsi-350000399b8900799 +mac_addr: + - 84:46:fe:73:a2:71 + - 84:46:fe:73:a2:72 + - 84:46:fe:73:a2:73 + - 84:46:fe:73:a2:74 + - 84:46:fe:73:a2:75 + - 84:46:fe:73:a2:76 + - 84:46:fe:73:a2:77 + - 84:46:fe:73:a2:78 + - 44:a1:91:35:1a:70 + - 44:a1:91:35:1a:71 + - 44:a1:91:35:1a:72 + - 44:a1:91:35:1a:73 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.12 -- 2.23.0

2 1

[PATCH v3 lkp-tests] feat(iso2rootfs): add step: submit rootfs test job after rootfs is made
by Yu Chuan 21 Oct '20

21 Oct '20

[Why] The produced dailybuild rootfs needs further testing by submit a job. [How] 1. Configure lkp env by {LKP_SRC}/sbin/install-dependencies.sh 2. Generate global compass-ci yaml. - The path of yaml: - /etc/compass-ci/defaults/sparrow.yaml - The yaml have two necessary fields: - SCHED_HOST - SCHED_PORT - The value of these two fields are consistent with the value in job.yaml of iso2rootfs. 3. Submit the job.yaml of rootfs test job with append follow params: - os - os_arch - os_version - os_mount - testbox Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 +++++ tests/iso2rootfs | 60 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index f413c9cee1f0..644c64be06b4 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,3 +11,9 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + test_yaml: + test_os: + test_os_arch: + test_os_version: + test_os_mount: + test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bf1f995e78df..85e3a73e8640 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,6 +53,12 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" + "test_yaml" + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" ) local yaml_t @@ -404,6 +410,57 @@ test_rootfs() EOF } +############ submit test job yaml ############ +generate_global_yaml() +{ + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + + mkdir -p "$(dirname "${config_yaml}")" + echo "SCHED_HOST: ${SCHED_HOST}" >> "${config_yaml}" + echo "SCHED_PORT: ${SCHED_PORT}" >> "${config_yaml}" +} + +generate_submit_append_str() +{ + local wait_add_vars=( + "os" + "os_arch" + "os_version" + "os_mount" + "testbox" + ) + for vt in "${wait_add_vars[@]}" + do + SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" + done +} + +submit_test_job_yaml() +{ + log_info "starting submit ${test_yaml} ..." + + [ -d "${LKP_SRC}" ] || die "no LKP_SRC !!!" + + cd "${LKP_SRC}" && { + chmod +x ./sbin/install-dependencies.sh + ./sbin/install-dependencies.sh + + generate_global_yaml + } + + cd "${LKP_SRC}/jobs" && { + [ -f "${test_yaml}" ] || + die "test yaml$${test_yaml}$ not exist in {LKP_SRC}/jobs !!!" + + generate_submit_append_str + [ -z "${SUBMIT_APPEND_STR}" ] || SUBMIT_APPEND_STR="-a ${SUBMIT_APPEND_STR}" + + "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + } + + log_info "submit ${test_yaml} finished" +} + ############ post works ############ post_works() { @@ -414,7 +471,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" + ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" @@ -433,6 +490,7 @@ main() run_qcow2rootfs test_rootfs + submit_test_job_yaml post_works } -- 2.23.0

1 0

[PATCH compass-ci] container/result-webdav: fix curl 403 forbidden error
by Cao Xueliang 21 Oct '20

21 Oct '20

[why] We cant find access_key_file in upload dest dir, because access_key_file was created in job result root dir, [how] 1. Add job id to the access_key. 2. Use job id to find the job result root dir in upload uri. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/nginx.conf | 8 +++++--- src/lib/job.cr | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 9297ae8..5ef3647 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -25,10 +25,12 @@ http { access_by_lua_block { accesskey = ngx.var.cookie_ACCESSKEY - uri=ngx.var.request_uri - path=string.match(uri,"%g*/") + job_id = string.sub(accesskey, string.find(accesskey,"-") - string.len(accesskey)) + uri = ngx.var.request_uri + path = string.match(uri,"%g*/") + root_path = string.sub(path, 1, string.find(path, job_id) - 1)..job_id.."/" - accesskey_file="/srv/"..path.."."..accesskey + accesskey_file = "/srv/"..root_path.."."..accesskey local f = io.open(accesskey_file) if not f then diff --git a/src/lib/job.cr b/src/lib/job.cr index 2cb8a2c..f6c1762 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -191,7 +191,7 @@ class Job end private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}" unless @hash["access_key"]? + self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" unless @hash["access_key"]? self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") end -- 2.23.0

1 0

[PATCH compass-ci] log_system/fluentd: change the architecture
by Wu Zhende 21 Oct '20

21 Oct '20

[Why] Now we have three types of fluentd:sub-fluentd/fluentd/master-fluentd. sub-fluentd and fluentd have similar functions, can be merged. So change to two types of fluentd:sub-fluentd and fluentd. sub-fluentd only collections logs then forward to fluentd. All data processing is performed on fluentd. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 86 +++++++------------- container/fluentd/start | 2 - container/master-fluentd/Dockerfile | 6 -- container/master-fluentd/build | 6 -- container/master-fluentd/build-depends | 1 - container/master-fluentd/docker-fluentd.conf | 48 ----------- container/master-fluentd/start | 26 ------ container/sub-fluentd/docker-fluentd.conf | 54 ++++++++++++ container/sub-fluentd/start | 12 +-- 9 files changed, 89 insertions(+), 152 deletions(-) delete mode 100644 container/master-fluentd/Dockerfile delete mode 100755 container/master-fluentd/build delete mode 100755 container/master-fluentd/build-depends delete mode 100644 container/master-fluentd/docker-fluentd.conf delete mode 100755 container/master-fluentd/start diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 23c7585..4660988 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -2,17 +2,9 @@ workers 10 </system> -<worker 0> <source> - @type tail - path /srv/cci/serial/logs/* - pos_file /srv/cci/serial/fluentd-pos/serial.log.pos - tag serial.* - path_key serial_path - refresh_interval 1s - <parse> - @type none - </parse> + @type forward + bind 0.0.0.0 </source> <filter **> @@ -24,34 +16,16 @@ </filter> <match serial.**> - @type copy - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> - </store> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> </match> -</worker> - -<worker 1-9> -<source> - @type forward - bind 0.0.0.0 -</source> - -<filter **> - @type record_transformer - enable_ruby - <record> - time ${time.strftime('%Y-%m-%d %H:%M:%S%z')} - </record> -</filter> <filter **> @type parser @@ -65,31 +39,29 @@ @type copy <store> - @type stdout - <format> - @type stdout - output_type single_value - message_key log - add_newline true - </format> + @type elasticsearch + host 172.17.0.1 + port 9202 + suppress_type_name true + flush_interval 1s + index_name ${tag} + ssl_verify false + log_es_400_reason true + with_transporter_log true </store> <store> - @type forward - flush_interval 0 - send_timeout 60 - heartbeat_type udp - heartbeat_interval 1 - recover_wait 10 - hard_timeout 60 - <server> - master-fluentd - host 172.17.0.1 - port 24225 - </server> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> </store> </match> -</worker> <label @FLUENT_LOG> <match fluent.*> diff --git a/container/fluentd/start b/container/fluentd/start index 8ad7c1e..b7f9c73 100755 --- a/container/fluentd/start +++ b/container/fluentd/start @@ -12,8 +12,6 @@ cmd = %w[ --restart=always --name fluentd -v /etc/localtime:/etc/localtime:ro - -v /srv/cci/serial/logs:/srv/cci/serial/logs:ro - -v /srv/cci/serial/fluentd-pos:/srv/cci/serial/fluentd-pos -d -u 1090:1090 -p 24224:24224/tcp diff --git a/container/master-fluentd/Dockerfile b/container/master-fluentd/Dockerfile deleted file mode 100644 index 192ece3..0000000 --- a/container/master-fluentd/Dockerfile +++ /dev/null @@ -1,6 +0,0 @@ -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -FROM fluentd-base:alpine - -COPY --chown=1090:1090 docker-fluentd.conf /fluentd/etc/docker-fluentd.conf diff --git a/container/master-fluentd/build b/container/master-fluentd/build deleted file mode 100755 index 7d0291d..0000000 --- a/container/master-fluentd/build +++ /dev/null @@ -1,6 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -system 'docker build -t master-fluentd:alpine .' diff --git a/container/master-fluentd/build-depends b/container/master-fluentd/build-depends deleted file mode 100755 index 29ee9d3..0000000 --- a/container/master-fluentd/build-depends +++ /dev/null @@ -1 +0,0 @@ -fluentd-base diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf deleted file mode 100644 index ef97d5c..0000000 --- a/container/master-fluentd/docker-fluentd.conf +++ /dev/null @@ -1,48 +0,0 @@ -<system> - workers 10 -</system> - -<source> - @type forward - bind 0.0.0.0 -</source> - -<match **> - @type copy - - <store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - log_es_400_reason true - with_transporter_log true - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> - </store> -</match> - -<label @FLUENT_LOG> - <match fluent.*> - @type stdout - <format> - @type stdout - output_type single_value - message_key message - add_newline true - </format> - </match> -</label> diff --git a/container/master-fluentd/start b/container/master-fluentd/start deleted file mode 100755 index 9e02980..0000000 --- a/container/master-fluentd/start +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -require_relative '../defconfig.rb' - -docker_rm "master-fluentd" - -cmd = %w[ - docker run - --restart=always - --name master-fluentd - -v /etc/localtime:/etc/localtime:ro - -d - -u 1090:1090 - -p 24225:24224/tcp - -p 24225:24224/udp - -e FLUENTD_CONF=docker-fluentd.conf - --log-driver json-file - --log-opt max-size=1g - master-fluentd:alpine -] -cmd += ['sh', '-c', 'umask 002 & fluentd -c /fluentd/etc/docker-fluentd.conf'] - -system(*cmd) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index 24ad2e6..0092982 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -1,3 +1,8 @@ +<system> + workers 10 +</system> + +<worker 0> <source> @type tail path /srv/cci/serial/logs/* @@ -26,3 +31,52 @@ overflow_action block </buffer> </match> +</worker> + +<worker 1-9> +<source> + @type forward + bind 0.0.0.0 +</source> + +<match **> + @type copy + + <store> + @type stdout + <format> + @type stdout + output_type single_value + message_key log + add_newline true + </format> + </store> + + <store> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host "#{ENV['FLUENTD_SERVER_HOST']}" + port "#{ENV['FLUENTD_SERVER_PORT']}" + </server> + </store> +</match> +</worker> + +<label @FLUENT_LOG> + <match fluent.*> + @type stdout + <format> + @type stdout + output_type single_value + message_key message + add_newline true + </format> + </match> +</label> diff --git a/container/sub-fluentd/start b/container/sub-fluentd/start index fa64dba..53a74c4 100755 --- a/container/sub-fluentd/start +++ b/container/sub-fluentd/start @@ -12,12 +12,13 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' +defaults['FLUENTD_SERVER_PORT'] ||= '24224' +env = docker_env(defaults) raise 'must config FLUENTD_SERVER_HOST' unless defaults['FLUENTD_SERVER_HOST'] docker_rm 'sub-fluentd' -cmd = %W[ +cmd = %w[ docker run --restart=always --name sub-fluentd @@ -25,12 +26,11 @@ cmd = %W[ -v /srv/cci/serial/logs:/srv/cci/serial/logs:ro -v /srv/cci/serial/fluentd-pos:/srv/cci/serial/fluentd-pos -d +] + env + %w[ -u 1090:1090 - -p 24224:24224/tcp - -p 24224:24224/udp + -p 24225:24224/tcp + -p 24225:24224/udp -e FLUENTD_CONF=docker-fluentd.conf - -e FLUENTD_SERVER_HOST=#{defaults['FLUENTD_SERVER_HOST']} - -e FLUENTD_SERVER_PORT=#{FLUENTD_SERVER_PORT} --log-driver json-file --log-opt max-size=1g sub-fluentd:alpine -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci