[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq
Signed-off-by: Wu Zhende wuzhende666@163.com --- container/fluentd/docker-fluentd.conf | 33 ++++++++++----------------- 1 file changed, 12 insertions(+), 21 deletions(-)
diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 8e76441..ac73dfb 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -60,28 +60,19 @@ </store>
<store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - num_threads 2 - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host 172.17.0.1 + port 24225 + </server> </store> - </match>
<label @FLUENT_LOG>