- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests 2/2] tests/iozone: add specified parameter for iozone test
by Lu Kaiyi 28 Oct '20

28 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for different block_size. [How] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..e69991fa 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,4 +1,9 @@ #!/bin/sh +# - block_size +# - file_size +# - write_rewrite +# - read_reread +# - random_read_write ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. @@ -7,7 +12,17 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + cmd="iozone" + if [ -n "$block_size" ]; then + cmd += " -r $block_size" + [ -n "$file_size" ] && cmd += " -s $file_size" + [ -n "$write_rewrite" ] && cmd += " -i $write_rewrite" + [ -n "$read_reread" ] && cmd += " -i $read_reread" + [ -n "$random_read_write" ] && cmd += " -i $random_read_write" + else + cmd += " -a" + fi + cmd += " -f $mnt/iozonetest" log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

3 3

[PATCH v2 compass-ci] sparrow/3-code: add centos in os file
by Zhang Yale 28 Oct '20

28 Oct '20

[Why] when run dev-env in centos system, error like this: ./dev-env:line 41: ./os/centos: No such file or directory. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- sparrow/3-code/os/centos | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 sparrow/3-code/os/centos diff --git a/sparrow/3-code/os/centos b/sparrow/3-code/os/centos new file mode 100644 index 0000000..b0669c0 --- /dev/null +++ b/sparrow/3-code/os/centos @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ + +cat > /etc/default/docker << EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF + +systemctl restart docker -- 2.23.0

1 0

[PATCH compass-ci] container/logging-es: optimize bulk write performance
by Wu Zhende 28 Oct '20

28 Oct '20

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 13 +++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..d32d10f 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a indices.memory.index_buffer_size: 30%' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..be9ac6d 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,16 @@ cmd=( ) "${cmd[@]}" + +echo "Please wait 60s for setting index" +sleep 60 + +curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{ + "index": { + "refresh_interval": "300s", + "translog.durability": "async", + "translog.sync_interval": "300s", + "translog.flush_threshold_size": "1024mb", + "merge.scheduler.max_thread_count": 1} +}' -- 2.23.0

1 0

[PATCH v3 compass-ci] container/debian: solve build error
by Cui Yili 28 Oct '20

28 Oct '20

[why] Executing "container/debian/build", error when apt-get installs the package: E: Could not configure 'libc6:arm64'. E: Could not perform immediate configuration on 'libnss-nis:arm64'. Please see man 5 apt.conf under APT::Immediate-Configure for details. (2) [how] Increase the parameter "-o APT::Immediate-Configure=false" when apt-get installs the package. reference: https://www.debian.org/releases/wheezy/armhf/release-notes/ch-upgrading.en.… Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/debian/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/debian/Dockerfile b/container/debian/Dockerfile index 1d310da..7f86cee 100644 --- a/container/debian/Dockerfile +++ b/container/debian/Dockerfile @@ -12,6 +12,6 @@ COPY root / RUN sed -i "s/^exit [0-9]*/exit 0/" /usr/sbin/policy-rc.d RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y runit openssh-server zsh vim rsync git make gcc g++ tzdata sudo && \ + apt-get install -y -o APT::Immediate-Configure=false runit openssh-server zsh vim rsync git make gcc g++ tzdata sudo && \ mkdir -p /run/sshd -- 2.23.0

1 0

[PATCH v2 lkp-tests] fix(distro/depends): add cci-depends/cci-makepkg file
by Wang Yong 28 Oct '20

28 Oct '20

[Why] prevent initramfs OS file from becoming too large [How] import packages from depends cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 2 ++ distro/depends/cci-makepkg | 10 ++++++++++ 2 files changed, 12 insertions(+) create mode 100644 distro/depends/cci-depends create mode 100644 distro/depends/cci-makepkg diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends new file mode 100644 index 000000000..1f7f73ac3 --- /dev/null +++ b/distro/depends/cci-depends @@ -0,0 +1,2 @@ +cifs-utils +cpio diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg new file mode 100644 index 000000000..a7ee67537 --- /dev/null +++ b/distro/depends/cci-makepkg @@ -0,0 +1,10 @@ +curl +libarchive-tools +bzip2 +fakeroot +gnupg +gettext +openssl +ncurses-bin +binutils +cifs-utils -- 2.23.0

2 2

[PATCH v5 compass-ci 1/2] add queue field into job.yaml
by Xiao Shenwei 28 Oct '20

28 Oct '20

queue field come from user's specify(job.yaml) or dafault Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/job.cr | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8f5dd36..7fbc414 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -73,6 +73,7 @@ class Job tbox_group testbox lab + queue initrd_pkg initrd_deps initrds_uri @@ -139,6 +140,7 @@ class Job set_kernel_params() set_lkp_server() set_sshr_port() + set_queue() end private def append_init_field @@ -199,6 +201,16 @@ class Job self["result_service"] = "raw_upload" end + private def set_queue + return if @hash.has_key?("queue") + + # set default value + self["queue"] = tbox_group + if tbox_group.to_s.starts_with?(/(vm|dc)-/) + self["queue"] = "#{tbox_group}.#{arch}" + end + end + # if not assign tbox_group, set it to a match result from testbox # ?if job special testbox, should we just set tbox_group=textbox private def update_tbox_group_from_testbox -- 2.23.0

1 1

[PATCH v2 compass-ci] specify the queues when the VM is started
by Xiao Shenwei 28 Oct '20

28 Oct '20

[why] specify the VM that need to consume the queues one job will added one queue, but one VM can consume job from multi queues. [how] to solve this problem, the scheduler should obtains which queues should be request. solution-1: register mac and queues relation before: mac=>hostname change to: mac=>hostname and mac=>queues then scheduler can get queues based on mac solution-2: /boot.ipxe/mac/${mac}?queues=xxx,yyy our HW and PXE-VM use /tftpboot/boot.ipxe to send request, unable to specify parameter, so use solution-1. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-qemu.sh | 4 +++- providers/qemu.sh | 12 ++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 954d3a0..7f3aa1b 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -4,6 +4,8 @@ [[ $tbox_group ]] || tbox_group=vm-2p8g -export hostname=$tbox_group--$USER-$$ +export hostname=$tbox_group.$USER-$$ +# specify which queues will be request, use " " to separate more than 2 values +export queues="vm-2p8g.aarch64 vm-2p8g~$USER" $CCI_SRC/providers/qemu.sh diff --git a/providers/qemu.sh b/providers/qemu.sh index d97fd85..6719e96 100755 --- a/providers/qemu.sh +++ b/providers/qemu.sh @@ -9,6 +9,7 @@ load_cci_defaults : ${hostname:="vm-1p1g-1"} +: ${queues:="vm-1p1g.$(arch)"} # unicast prefix: x2, x6, xA, xE export mac=$(echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') echo hostname: $hostname @@ -17,7 +18,14 @@ echo $mac > mac echo "arp -n | grep ${mac//-/:}" > ip.sh chmod +x ip.sh -curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" +set_mac2host() +{ + # use "," replace " " + queues=$(echo $queues | sed -r 's/ +/,/g') + echo $queues + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}&queues=${queues}" +} +set_mac2host del_host_mac() { @@ -33,7 +41,7 @@ trap del_host_mac EXIT tbox_group=$hostname fi - host=${tbox_group%%--*} + host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

2 3

[PATCH v3 lkp-tests 1/2] jobs/iozone_with_block_size.yaml: add iozone_with_block_size.yaml
by Lu Kaiyi 28 Oct '20

28 Oct '20

[why] need test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone_with_block_size.yaml | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 jobs/iozone_with_block_size.yaml diff --git a/jobs/iozone_with_block_size.yaml b/jobs/iozone_with_block_size.yaml new file mode 100644 index 00000000..f3937e9e --- /dev/null +++ b/jobs/iozone_with_block_size.yaml @@ -0,0 +1,27 @@ +suite: iozone +testcase: iozone +category: benchmark + +file_size: 4g +write_rewrite: 0 +read_reread: 1 +random_read_write: 2 + +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + +disk: 2HDD + +fs: +- xfs +- ext4 +- btrfs + +iosched: +- kyber + +iozone: -- 2.23.0

1 0

[PATCH lkp-tests] fix(distro/depends): add cci-depends/cci-makepkg file
by Wang Yong 28 Oct '20

28 Oct '20

[Why] prevent initramfs OS file from becoming too large [How] import packages from depends cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 2 ++ distro/depends/cci-makepkg | 9 +++++++++ 2 files changed, 11 insertions(+) create mode 100644 distro/depends/cci-depends create mode 100644 distro/depends/cci-makepkg diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends new file mode 100644 index 000000000..1f7f73ac3 --- /dev/null +++ b/distro/depends/cci-depends @@ -0,0 +1,2 @@ +cifs-utils +cpio diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg new file mode 100644 index 000000000..aecd50e0f --- /dev/null +++ b/distro/depends/cci-makepkg @@ -0,0 +1,9 @@ +curl +libarchive-tools +bzip2 +fakeroot +gnupg +gettext +openssl +ncurses-bin +binutils -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] based on queue field add the job into specify queue
by Xiao Shenwei 28 Oct '20

28 Oct '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 692180f..b66c19d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -30,12 +30,14 @@ class Sched mac.gsub(":", "-") end - def set_host_mac(mac : String, hostname : String) + def set_host_mac(mac : String, hostname : String, queues : String) @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + @redis.hash_set("sched/mac2queues", normalize_mac(mac), queues) end def del_host_mac(mac : String) @redis.hash_del("sched/mac2host", normalize_mac(mac)) + @redis.hash_del("sched/mac2queues", normalize_mac(mac)) end # return: @@ -251,24 +253,24 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - tbox_group = job.tbox_group + queue = job.queue return [{ "job_id" => "0", - "message" => "get tbox group failed", + "message" => "get queue failed", "job_state" => "submit", - }] unless tbox_group + }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? - tbox_group += "#{job.get_uuid_tag}" + queue += "#{job.get_uuid_tag}" else - tbox_group = "#{tbox_group}/idle" + queue = "#{queue}/idle" end - job_id = add_task(tbox_group, job.lab) + job_id = add_task(queue, job.lab) return [{ "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", }] unless job_id @@ -372,7 +374,7 @@ class Sched hostname = @redis.hash_get("sched/mac2host", normalize_mac(api_param)) if hostname.nil? # auto name new/unknown machine hostname = "sut-#{api_param}" - set_host_mac(api_param, hostname) + set_host_mac(api_param, hostname, "") # auto submit a job to collect the host information # grub hostname is link with ":", like "00:01:02:03:04:05" -- 2.23.0

1 0