November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] area: fix src/lib/web_backend.rb "/compare_candidates" tbox_group regex error
by Zhang Yuhang 02 Nov '20

02 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2

[PATCH v4 compass-ci] container/logging-es: optimize bulk write performance
by Wu Zhende 02 Nov '20

02 Nov '20

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios. Before: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.116s, query:0.114s] After: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.060s, query:0.057s] [How] 1.Increase index.refresh_interval 2.Set index.translog.durability to Async 3.Increase the JVM memory 4.Increase write.queue_size 5.Increase index_buffer_size [Reference Link] https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-ind… https://ravishajava.com/2018/07/13/elastic-search-performance-tuning-parmet… https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-al… https://thehoard.blog/techniques-to-achieve-high-write-throughput-with-elas… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 12 ++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..707299f 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..25ed36b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,15 @@ cmd=( ) "${cmd[@]}" + +echo "Please wait 60s for setting index" +sleep 60 + +curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{ + "index": { + "refresh_interval": "120s", + "translog.durability": "async", + "translog.sync_interval": "120s", + "translog.flush_threshold_size": "1024mb"} +}' -- 2.23.0

1 0

[PATCH v7 compass-ci 1/5] container/ntp-server: Dockerfile
by Wang Chenglong 01 Nov '20

01 Nov '20

[What] The docker file for build a container to provide network time synchronization service for virtual and physical machines. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/ntp-server/Dockerfile diff --git a/container/ntp-server/Dockerfile b/container/ntp-server/Dockerfile new file mode 100644 index 0000000..76f2a0e --- /dev/null +++ b/container/ntp-server/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos:7 +MAINTAINER Wang Chenglong <18509160991(a)163.com> + +RUN yum repolist +RUN yum -y install ntp && \ + yum clean all +RUN ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + +COPY ntp.conf /etc/ntp.conf +COPY start-server.sh / + +EXPOSE 123 + +ENTRYPOINT ["/start-server.sh"] +CMD ["sh"] -- 2.23.0

3 3

[PATCH v5 lkp-tests 2/2] tests/iozone: add specified parameter for iozone test
by Lu Kaiyi 01 Nov '20

01 Nov '20

[why] need to test write, rewrite, read, reread, random write, random read for different block_size. [how] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..88a92a18 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,13 +1,49 @@ #!/bin/sh - +# - block_size +# - file_size +# - write_rewrite +# - read_reread +# - random_read_write +# - read_backwards +# - rewrite_record +# - stride_read +# - fwrite_refwrite +# - fread_refread +# - random_mix +# - pwrite_repwrite +# - pread_repread +# - pwritev_repwritev +# - preadv_repreadv ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh + +args="iozone" +if [ -n "$block_size" ]; then + args+=" -r $block_size" + [ -n "$file_size" ] && args+=" -s $file_size" + [ -n "$write_rewrite" ] && args+=" -i 0" + [ -n "$read_reread" ] && args+=" -i 1" + [ -n "$random_read_write" ] && args+=" -i 2" + [ -n "$read_backwards" ] && args+=" -i 3" + [ -n "$rewrite_record" ] && args+=" -i 4" + [ -n "$stride_read" ] && args+=" -i 5" + [ -n "$fwrite_refwrite" ] && args+=" -i 6" + [ -n "$fread_refread" ] && args+=" -i 7" + [ -n "$random_mix" ] && args+=" -i 8" + [ -n "$pwrite_repwrite" ] && args+=" -i 9" + [ -n "$pread_repread" ] && args+=" -i 10" + [ -n "$pwritev_repwritev" ] && args+=" -i 11" + [ -n "$preadv_repreadv" ] && args+=" -i 12" +else + args+=" -a" +fi + for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + cmd="${args} -f $mnt/iozonetest" log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

2 1