For optimization patch, better show case the performance numbers before/after the patch.

On Wed, Oct 28, 2020 at 04:06:31PM +0800, Wu Zhende wrote:

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios.

I find that information not very useful to help me understand the change. Do you referenced some url article to set the parameters? Please say something 有具体指导性的。万一以后要继续调整呢？ 那我最好要知道之前的参数是怎么来的。

Thanks, Fengguang

Signed-off-by: Wu Zhende wuzhende666@163.com

container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 12 ++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..7199cba 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2

MAINTAINER Wu Zhende wuzhende666@163.com

-RUN sed -i 's:#network.host: _site_:network.host: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host: _site_:network.host: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \

• sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \
• sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \
• sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \
• sed -i '$a indices.memory.index_buffer_size: 30%' /usr/share/elasticsearch/config/elasticsearch.yml && \
• sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \
• sed -i "s/-Xms1g/-Xms32g/g" /usr/share/elasticsearch/config/jvm.options && \
• sed -i "s/-Xmx1g/-Xmx32g/g" /usr/share/elasticsearch/config/jvm.options

RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..25ed36b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,15 @@ cmd=( )

"${cmd[@]}"

+echo "Please wait 60s for setting index" +sleep 60

+curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{

• "index": {
• "refresh_interval": "120s",
• "translog.durability": "async",
• "translog.sync_interval": "120s",
• "translog.flush_threshold_size": "1024mb"}

+}'

2.23.0