- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] bin/lkp-setup-rootfs: fix auto submit makepkg task
by Li Ping 30 Oct '20

30 Oct '20

[why] the define_files will generate the PKGBUILD and when run define_files_auto_pack will automatic run makepkg even though the job is not for makepkg --- bin/lkp-setup-rootfs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 2890601e..a3663b42 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -187,14 +187,14 @@ define_files_pkg_pack() ( local benchmark=${1#pkg/} export benchmark=${benchmark%/PKGBUILD} - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) } # There may be $program-dev or $program-dev-dev in the # "$LKP_SRC/distro/depends" directory, # when the value passed by $define_files is "distro/depends/$program-dev", -# you don't know which script in "makepkg" or "pack-deps" to execute, +# you don't know which script in "cci-makepkg" or "pack-deps" to execute, # so it needs to be handled sparately. # Save the result to # "/srv/initrd/deps/$os/$os_arch/$os_version/$deps_benchmark.cgz", @@ -208,7 +208,7 @@ define_files_deps_pack() [ $suffix == "dev" ] && [ -f "$LKP_SRC/pkg/$pkg_benchmark/PKGBUILD" ] && { ( export benchmark=$pkg_benchmark - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) return 0 } @@ -228,10 +228,10 @@ define_files_auto_pack() do [ -f "$LKP_SRC/$path" ] || continue - [ ${path##*/} = PKGBUILD ] && { - define_files_pkg_pack $path + [ ${path##*/} = PKGBUILD ] && [ $suite = 'cci-makepkg' ] && { + define_files_pkg_pack $path } - [ ${path#distro/depends/} != $path ] && { + [ ${path#distro/depends/} != $path ] && [ $suite = 'cci-depends' ] && { define_files_deps_pack $path } done -- 2.23.0

2 1

[PATCH v6 compass-ci 5/5] container/ntp-server: start
by Wang Chenglong 30 Oct '20

30 Oct '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/start | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 container/ntp-server/start diff --git a/container/ntp-server/start b/container/ntp-server/start new file mode 100755 index 0000000..c6f04a6 --- /dev/null +++ b/container/ntp-server/start @@ -0,0 +1,17 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +SCRIPT_DIR=$(dirname $(realpath $0)) +. $(dirname $SCRIPT_DIR)/defconfig.sh + +docker_rm ntp-server +cmd=( + docker run + -id + --name=ntp-server + -p 123:123/udp + ntp-server +) + +"${cmd[@]}" -- 2.23.0

3 4

[PATCH v3 lkp-tests] fix: btrfs-progs is not installed when os_mount=initramfs
by Hu Xuejiao 29 Oct '20

29 Oct '20

[why] mkfs.btrfs "command not found" when os_mount=initramfs [how] it will user mkfs.ext4 because mkfs.ext4 is common, other mkfs.*** is not considered [how] mkfs.btrfs and mkfs.ext4 is no exist, it will return Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/bootstrap.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d68..9744cb84 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -456,7 +456,12 @@ mount_rootfs() ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { - mkfs.btrfs -f $rootfs_partition + if has_cmd mkfs.btrfs; then + mkfs.btrfs -f $rootfs_partition + else + mkfs.ext4 -F $rootfs_partition || return + fi + mount $rootfs_partition $ROOTFS_DIR } mkdir -p $ROOTFS_DIR/tmp -- 2.23.0

3 2

[PATCH v2 lkp-tests] hackbench/PKGBUILD: fix source and build function
by Zhang Yale 29 Oct '20

29 Oct '20

[Why] Old source "https://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-1.8.tar.gz" is not available, and need build before package. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- pkg/hackbench/PKGBUILD | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/pkg/hackbench/PKGBUILD b/pkg/hackbench/PKGBUILD index 797a00e6..7e3b52e9 100644 --- a/pkg/hackbench/PKGBUILD +++ b/pkg/hackbench/PKGBUILD @@ -4,10 +4,17 @@ pkgrel=1 arch=('i386' 'x86_64') url="https://www.kernel.org/pub/linux/utils/rt-tests" license=('GPL') -source=("https://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-${pkgver}.tar.gz") +source=("https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/snapshot/rt-test…") sha256sums=('SKIP') -package() { +build() +{ + cd "$srcdir/rt-tests-${pkgver}" + gcc -o hackbench hackbench.c -lpthread +} + +package() +{ mkdir -p "${pkgdir}/lkp/benchmarks/${pkgname}" cd "$srcdir/rt-tests-${pkgver}" make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" -- 2.23.0

3 4

[PATCH v2 compass-ci] save hostname and queues relation in redis
by Xiao Shenwei 29 Oct '20

29 Oct '20

[why] the test machine has the ability to know which queues should be requested [how] for VM HW, we will register mac2hostname and hostname2queues. then they can find queues based on mac. API: boot.ipxe/mac/${mac} for docker, we will register hostname2queues, then container can find queues based on hostname. API: boot.container/hostname/${hostname} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 8 ++++++++ src/scheduler/scheduler.cr | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b2ef6ce..9407ddc 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -34,10 +34,18 @@ class Sched @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) end + def set_host2queues(hostname : String, queues : String) + @redis.hash_set("sched/host2queues", hostname, queues) + end + def del_host_mac(mac : String) @redis.hash_del("sched/mac2host", normalize_mac(mac)) end + def del_host2queues(hostname : String) + @redis.hash_del("sched/host2queues", hostname) + end + # return: # Hash(String, Hash(String, String)) def get_cluster_state(cluster_id) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 371e4ed..a946ea0 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -113,6 +113,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" + put "/set_host2queues" do |env| + env.response.headers["Connection"] = "close" + if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) + sched.set_host2queues(client_host, client_queues) + + "Done" + else + "No yet" + end + end + # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| env.response.headers["Connection"] = "close" @@ -125,6 +137,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" + put "/del_host2queues" do |env| + env.response.headers["Connection"] = "close" + if client_host = env.params.query["host"]? + sched.del_host2queues(client_host) + + "Done" + else + "No yet!" + end + end + # client(runner) report job's status # /~lkp/cgi-bin/lkp-jobfile-append-var # ?job_file=/lkp/scheduled/job.yaml&job_state=running&job_id=10 -- 2.23.0

3 4

[PATCH v2 lkp-tests] fix: btrfs-progs is not installed when os_mount=initramfs
by Hu Xuejiao 29 Oct '20

29 Oct '20

[why] mkfs.btrfs "command not found" when os_mount=initramfs [how] it will user mkfs.ext4. because mkfs.ext4 is common, other mkfs.*** is not considered [how] mkfs.btrf and mkfs.ext4 is no exist, it will return Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/bootstrap.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d68..9744cb84 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -456,7 +456,12 @@ mount_rootfs() ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { - mkfs.btrfs -f $rootfs_partition + if has_cmd mkfs.btrfs; then + mkfs.btrfs -f $rootfs_partition + else + mkfs.ext4 -F $rootfs_partition || return + fi + mount $rootfs_partition $ROOTFS_DIR } mkdir -p $ROOTFS_DIR/tmp -- 2.23.0

3 2

[PATCH compass-ci] container/scheduler/start: fix duplicate -e
by Wu Zhende 29 Oct '20

29 Oct '20

[Why] "env = docker_env(defaults)" already add "-e xxx". Don't need to add in cmd again. SCHED_HOST/SCHED_PORT/GIT_SERVER/MASTER_FLUENTD_HOST/MASTER_FLUENTD_HOST may add duplicate -e. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index 6898d4b..70a3496 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -32,15 +32,17 @@ defaults = relevant_defaults(names) defaults['SSHR_PORT'] ||= 5050 defaults['SSHR_PORT_BASE'] ||= 50000 defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -MASTER_FLUENTD_PORT = defaults['MASTER_FLUENTD_PORT'] || '24224' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -61,11 +63,6 @@ cmd = %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} - -e GIT_SERVER=#{defaults["GIT_SERVER"]} - -e MASTER_FLUENTD_HOST=#{defaults["MASTER_FLUENTD_HOST"]} - -e MASTER_FLUENTD_PORT=#{MASTER_FLUENTD_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -- 2.23.0

1 0

[PATCH lkp-tests] tests: add get_cpu function
by Zhang Yu 29 Oct '20

29 Oct '20

[why] Obtain more host information from 'lscpu' Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/tests/host-info b/tests/host-info index 46f43f93..84f796dc 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,18 +1,25 @@ #!/bin/bash -get_node() +get_memory() { - echo "nr_node: $(grep -Eo 'Node [0-9]+,' /proc/zoneinfo | uniq | wc -l)" - echo "nr_cpu: $(nproc)" memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G echo "memory: $memory" } -get_model_name() +get_cpu() { - echo "model_name: $(lscpu |grep "Model name" | awk '{print $3}')" + echo $1: $(lscpu | awk -F': *' '/'$2':/{print $2}') +} + +get_cpu_info() +{ + cpu_info=("arch Architecture" "nr_node NUMA\snode$s$" "nr_cpu ^CPU$s$" "model_name Model\sname") + for info in "${cpu_info[@]}" + do + get_cpu $info + done } get_nr_disk() @@ -101,11 +108,11 @@ get_ipmi_ip() main() { - get_node + get_memory get_nr_disk get_partitions get_mac - get_model_name + get_cpu_info get_ipmi_ip } -- 2.23.0

3 4

[PATCH v3 compass-ci] fix: sched: add 2 fields into cluster state
by Ren Wen 29 Oct '20

29 Oct '20

[Why] 1) Each node will report it's infos (roles, ip, direct_ips, direct_macs) when running cluster job, so also add these 2 parameters into node state: "direct_ips", "direct_macs". Cluster test scripts needs "direct_server_ips" as server client's ip to connect to server to do a test. When request state is "roles_ip", return clients two variables: server, direct_server_ips. Clients will export them. 2) Alter funtion "get_ip"'s name and it's return value to fit for changes. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index c32091f..b981876 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -99,24 +99,31 @@ class Sched when "write_state" node_roles = env.params.query["node_roles"] node_ip = env.params.query["ip"] + direct_ips = env.params.query["direct_ips"] + direct_macs = env.params.query["direct_macs"] + update_cluster_state(cluster_id, job_id, "roles", node_roles) update_cluster_state(cluster_id, job_id, "ip", node_ip) + update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) + update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) when "roles_ip" role = "server" - server_ip = get_ip(cluster_id, role) - return "server=#{server_ip}" + role_state = get_role_state(cluster_id, role) + raise "Missing #{role} state in cluster state" unless role_state + return "server=#{role_state["ip"]}\n" \ + "direct_server_ips=#{role_state["direct_ips"]}" end # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) end - # get the ip of role from cluster_state - def get_ip(cluster_id, role) + # get the node state of role from cluster_state + private def get_role_state(cluster_id, role) cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |config| - if %(#{config["roles"]}) == role - return config["ip"] + cluster_state.each_value do |role_state| + if %(#{role_state["roles"]}) == role + return role_state end end end -- 2.23.0

6 12

[PATCH v3 compass-ci] container/logging-es: optimize bulk write performance
by Wu Zhende 29 Oct '20

29 Oct '20

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios. Before: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.116s, query:0.114s] After: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.060s, query:0.057s] [How] 1.Increase index.refresh_interval 2.Set index.translog.durability to Async 3.Increase the JVM memory 4.Increase write.queue_size 5.Increase index_buffer_size [Reference Link] https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-ind… https://ravishajava.com/2018/07/13/elastic-search-performance-tuning-parmet… https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-al… https://thehoard.blog/techniques-to-achieve-high-write-throughput-with-elas… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 12 ++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..707299f 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..25ed36b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,15 @@ cmd=( ) "${cmd[@]}" + +echo "Please wait 60s for setting index" +sleep 60 + +curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{ + "index": { + "refresh_interval": "120s", + "translog.durability": "async", + "translog.sync_interval": "120s", + "translog.flush_threshold_size": "1024mb"} +}' -- 2.23.0

1 0