October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests: add get_cpu function
by Zhang Yu 29 Oct '20

29 Oct '20

[why] Obtain more host information from 'lscpu' Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/tests/host-info b/tests/host-info index 46f43f93..84f796dc 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,18 +1,25 @@ #!/bin/bash -get_node() +get_memory() { - echo "nr_node: $(grep -Eo 'Node [0-9]+,' /proc/zoneinfo | uniq | wc -l)" - echo "nr_cpu: $(nproc)" memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G echo "memory: $memory" } -get_model_name() +get_cpu() { - echo "model_name: $(lscpu |grep "Model name" | awk '{print $3}')" + echo $1: $(lscpu | awk -F': *' '/'$2':/{print $2}') +} + +get_cpu_info() +{ + cpu_info=("arch Architecture" "nr_node NUMA\snode$s$" "nr_cpu ^CPU$s$" "model_name Model\sname") + for info in "${cpu_info[@]}" + do + get_cpu $info + done } get_nr_disk() @@ -101,11 +108,11 @@ get_ipmi_ip() main() { - get_node + get_memory get_nr_disk get_partitions get_mac - get_model_name + get_cpu_info get_ipmi_ip } -- 2.23.0

3 4

[PATCH v3 compass-ci] fix: sched: add 2 fields into cluster state
by Ren Wen 29 Oct '20

29 Oct '20

[Why] 1) Each node will report it's infos (roles, ip, direct_ips, direct_macs) when running cluster job, so also add these 2 parameters into node state: "direct_ips", "direct_macs". Cluster test scripts needs "direct_server_ips" as server client's ip to connect to server to do a test. When request state is "roles_ip", return clients two variables: server, direct_server_ips. Clients will export them. 2) Alter funtion "get_ip"'s name and it's return value to fit for changes. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index c32091f..b981876 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -99,24 +99,31 @@ class Sched when "write_state" node_roles = env.params.query["node_roles"] node_ip = env.params.query["ip"] + direct_ips = env.params.query["direct_ips"] + direct_macs = env.params.query["direct_macs"] + update_cluster_state(cluster_id, job_id, "roles", node_roles) update_cluster_state(cluster_id, job_id, "ip", node_ip) + update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) + update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) when "roles_ip" role = "server" - server_ip = get_ip(cluster_id, role) - return "server=#{server_ip}" + role_state = get_role_state(cluster_id, role) + raise "Missing #{role} state in cluster state" unless role_state + return "server=#{role_state["ip"]}\n" \ + "direct_server_ips=#{role_state["direct_ips"]}" end # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) end - # get the ip of role from cluster_state - def get_ip(cluster_id, role) + # get the node state of role from cluster_state + private def get_role_state(cluster_id, role) cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |config| - if %(#{config["roles"]}) == role - return config["ip"] + cluster_state.each_value do |role_state| + if %(#{role_state["roles"]}) == role + return role_state end end end -- 2.23.0

6 12

[PATCH v3 compass-ci] container/logging-es: optimize bulk write performance
by Wu Zhende 29 Oct '20

29 Oct '20

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios. Before: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.116s, query:0.114s] After: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.060s, query:0.057s] [How] 1.Increase index.refresh_interval 2.Set index.translog.durability to Async 3.Increase the JVM memory 4.Increase write.queue_size 5.Increase index_buffer_size [Reference Link] https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-ind… https://ravishajava.com/2018/07/13/elastic-search-performance-tuning-parmet… https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-al… https://thehoard.blog/techniques-to-achieve-high-write-throughput-with-elas… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 12 ++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..707299f 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..25ed36b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,15 @@ cmd=( ) "${cmd[@]}" + +echo "Please wait 60s for setting index" +sleep 60 + +curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{ + "index": { + "refresh_interval": "120s", + "translog.durability": "async", + "translog.sync_interval": "120s", + "translog.flush_threshold_size": "1024mb"} +}' -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] jobs/iozone_with_block_size.yaml: add iozone_with_block_size.yaml
by Lu Kaiyi 29 Oct '20

29 Oct '20

[why] need test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone_with_block_size.yaml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 jobs/iozone_with_block_size.yaml diff --git a/jobs/iozone_with_block_size.yaml b/jobs/iozone_with_block_size.yaml new file mode 100644 index 00000000..fb7933d8 --- /dev/null +++ b/jobs/iozone_with_block_size.yaml @@ -0,0 +1,25 @@ +suite: iozone +testcase: iozone +category: benchmark + +file_size: 4g +write_rewrite: 0 +read_reread: 1 +random_read_write: 2 + +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + +disk: 1HDD + +fs: +- xfs + +iosched: +- kyber + +iozone: -- 2.23.0

4 5

[PATCH lkp-tests] setup/sysctl: add sysctl params for nginx test
by Wei Jihui 29 Oct '20

29 Oct '20

nginx test need optimize network so add these parameters to setup/sysctl Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/sysctl | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 8b7d8339..81374939 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -193,6 +193,15 @@ # - sc_hot_threshold_mbytes # - sc_cold_threshold_mbytes # - sc_drop_threshold_mbytes +# - sc_ipv4_tcp_tw_reuse +# - sc_ipv4_tcp_keepalive_time +# - sc_ipv4_tcp_fin_timeout +# - sc_ipv4_tcp_max_tw_buckets +# - sc_ipv4_tcp_max_syn_backlog +# - sc_ipv4_ip_local_port_range +# - sc_netdev_max_backlog +# - sc_rmem_max +# - sc_wmem_max . $LKP_SRC/lib/reproduce-log.sh @@ -390,5 +399,14 @@ [ -n "$sc_hot_threshold_mbytes" ] && log_eval "echo '$sc_hot_threshold_mbytes' > '/proc/sys/vm/hot_threshold_mbytes'" [ -n "$sc_cold_threshold_mbytes" ] && log_eval "echo '$sc_cold_threshold_mbytes' > '/proc/sys/vm/cold_threshold_mbytes'" [ -n "$sc_drop_threshold_mbytes" ] && log_eval "echo '$sc_drop_threshold_mbytes' > '/proc/sys/vm/drop_threshold_mbytes'" +[ -n "$sc_ipv4_tcp_tw_reuse" ] && log_eval "echo '$sc_ipv4_tcp_tw_reuse' > '/proc/sys/net/ipv4/tcp_tw_reuse'" +[ -n "$sc_ipv4_tcp_keepalive_time" ] && log_eval "echo '$sc_ipv4_tcp_keepalive_time' > '/proc/sys/net/ipv4/tcp_keepalive_time'" +[ -n "$sc_ipv4_tcp_fin_timeout" ] && log_eval "echo '$sc_ipv4_tcp_fin_timeout' > '/proc/sys/net/ipv4/tcp_fin_timeout'" +[ -n "$sc_ipv4_tcp_max_tw_buckets" ] && log_eval "echo '$sc_ipv4_tcp_max_tw_buckets' > '/proc/sys/net/ipv4/tcp_max_tw_buckets'" +[ -n "$sc_ipv4_tcp_max_syn_backlog" ] && log_eval "echo '$sc_ipv4_tcp_max_syn_backlog' > '/proc/sys/net/ipv4/tcp_max_syn_backlog'" +[ -n "$sc_ipv4_ip_local_port_range" ] && log_eval "echo '$sc_ipv4_ip_local_port_range' > '/proc/sys/net/ipv4/ip_local_port_range'" +[ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" +[ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" +[ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" exit 0 -- 2.23.0

4 3

[PATCH v6 compass-ci 4/5] container/ntp-server: ntp.conf
by Wang Chenglong 29 Oct '20

29 Oct '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/ntp.conf | 60 +++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 container/ntp-server/ntp.conf diff --git a/container/ntp-server/ntp.conf b/container/ntp-server/ntp.conf new file mode 100644 index 0000000..67dd1b0 --- /dev/null +++ b/container/ntp-server/ntp.conf @@ -0,0 +1,60 @@ +# For more information about this file, see the ntp.conf(5) man page. + +# Record the frequency of the system clock. +driftfile /var/lib/ntp/drift + +# Permit time synchronization with our time source, but do not +# permit the source to query or modify the service on this system. +restrict default nomodify notrap nopeer noepeer noquery + +# Permit association with pool servers. +restrict source nomodify notrap noepeer noquery + +# Permit all access over the loopback interface. This could +# be tightened as well, but to do so would effect some of +# the administrative functions. +restrict 127.0.0.1 +restrict ::1 + +# Hosts on local network are less restricted. +restrict 172.0.0.0 mask 255.0.0.0 nomodify notrap + +# Use public servers from the pool.ntp.org project. +# Please consider joining the pool (http://www.pool.ntp.org/join.html) +# pool 2.openEuler.pool.ntp.org iburst +server 0.centos.pool.ntp.org +server 0.asia.pool.ntp.org +server ntp.sjtu.edu.cn +server ntp1.aliyun.com + +# When all ntp servers on Internet are inaccessible, it will use +# it's local time to give service to the ntp clients. +# If not set, the ntp server will go to stratum 16 and cannot serve +# any ntp clients when it loses connection with upstream ntp servers. +# This is common in a LAN environment isolated with Internet. +server 127.127.1.0 +fudge 127.127.1.0 stratum 8 + +# Reduce the maximum number of servers used from the pool. +tos maxclock 5 + +# Enable public key cryptography. +# crypto + +includefile /etc/ntp/crypto/pw + +# Key file containing the keys and key identifiers used when operating +# with symmetric key cryptography. +keys /etc/ntp/keys + +# Specify the key identifiers which are trusted. +# trustedkey 4 8 42 + +# Specify the key identifier to use with the ntpdc utility. +# requestkey 8 + +# Specify the key identifier to use with the ntpq utility. +# controlkey 8 + +# Enable writing of statistics records. +# statistics clockstats cryptostats loopstats peerstats -- 2.23.0

1 0

[PATCH v6 compass-ci 3/5] container/ntp-server: build
by Wang Chenglong 29 Oct '20

29 Oct '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/ntp-server/build diff --git a/container/ntp-server/build b/container/ntp-server/build new file mode 100755 index 0000000..7826fb2 --- /dev/null +++ b/container/ntp-server/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t ntp-server . -- 2.23.0

1 0

[PATCH v6 compass-ci 2/5] container/ntp-server: start-server.sh
by Wang Chenglong 29 Oct '20

29 Oct '20

[What] start ntp-server and specify the log file. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/start-server.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 container/ntp-server/start-server.sh diff --git a/container/ntp-server/start-server.sh b/container/ntp-server/start-server.sh new file mode 100755 index 0000000..bf87bd5 --- /dev/null +++ b/container/ntp-server/start-server.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +/usr/sbin/ntpd -g -l /var/log/ntpstats/ntpd.log + +exec "$@" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: fix no such file or directory
by Zhang Yale 29 Oct '20

29 Oct '20

[Why] when run dev-env in centos system error like this: ./dev-env:line 41: ./os/centos: No such file or directory. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 11c7fb0..7703786 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -38,4 +38,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +[ -f "${ID}" ] && . $(dirname ${BASH_SOURCE[0]})/os/${ID} -- 2.23.0

3 4

[PATCH v2 compass-ci] container/rsync-server: modify data upload directory
by Cui Yili 29 Oct '20

29 Oct '20

[why] Modify data upload directory to upload qemu image. qemu image uploaded by the user, used for testing [how] Change the rsync.conf configuration and docker operating parameters. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/rsync-server/rsyncd.conf | 6 ++++-- container/rsync-server/start | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index 528e8bd..dd8f128 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -1,10 +1,12 @@ uid = lkp gid = lkp +port = 11387 use chroot = no +log file=/var/log/rsync.log -[result] -path = /result +[upload] +path = /srv/initrd/qemu-image read only = no max connections = 200 refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 4124192..6fa3de2 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -9,9 +9,9 @@ cmd=( docker run -id --name=rsync_server - -v /srv/result:/result + -v /srv/initrd/qemu-image:/srv/initrd/qemu-image -v /etc/localtime:/etc/localtime:ro - -p 873:873 + -p 11387:11387 rsync_server:latest ) -- 2.23.0

3 2