January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] feat: scheduler: add new boot way
by Ren Wen 06 Jan '21

06 Jan '21

A new way to find job, requested by: curl -X GET 'http://<host>:<port>/boot.libvirt/mac/<mac_value>' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..f6981e7 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -135,6 +135,29 @@ class Sched return response.to_json end + private def get_boot_libvirt(job : Job) + response = {} of String => String | Nil | Array(String) | Hash(String, String) + + response["job_id"] = job.id + response["kernel_uri"] = job.kernel_uri.lchop("kernel ") + + initrds_uri = job["initrds_uri"]? + initrds_uri = initrds_uri.as_a.map(&.as_s.lchop("initrd ")) if initrds_uri + response["initrds_uri"] = initrds_uri + + kernel_params = job["kernel_params"]? + kernel_params = kernel_params.as_a.map(&.to_s).reject(&.starts_with?("initrd=")).join(" ") if kernel_params + response["kernel_params"] = kernel_params + + response["result_root"] = job.result_root + response["LKP_SERVER"] = "#{job["LKP_SERVER"]?}" + + # Not implement now + response["templates"] = {} of String => String + + return response.to_json + end + private def get_boot_grub(job : Job) initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" @@ -176,6 +199,8 @@ class Sched case boot_type when "ipxe" return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "libvirt" + return job ? get_boot_libvirt(job) : {"info" => "No job now"}.to_json when "grub" return job ? get_boot_grub(job) : grub_msg("No job now") when "container" -- 2.23.0

1 1

[PATCH lkp-tests] lib/detect-system.sh: add system arch riscv64
by Liu Yinsi 06 Jan '21

06 Jan '21

[why] when run job in riscv64 qemu, _system_arch=unknown, it cause qemu quit, so add it. test: /srv/result/iperf/2021-01-06/vm-8p64g/debian-sid-riscv64/tcp-30/z9.281254 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/detect-system.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/detect-system.sh b/lib/detect-system.sh index 90006c8f0..1b344b4f0 100755 --- a/lib/detect-system.sh +++ b/lib/detect-system.sh @@ -31,6 +31,8 @@ parse_executable_arch() _system_arch=aarch64;; *ARM) _system_arch=arm;; + *RISC-V) + _system_arch=riscv64;; *) _system_arch=unknown return 1;; -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: add wget in initramfs-common
by Zhang Yale 06 Jan '21

06 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/initramfs-common | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common index 5d3ecb76..5f766da6 100644 --- a/distro/depends/initramfs-common +++ b/distro/depends/initramfs-common @@ -1,3 +1,4 @@ time +wget perf which -- 2.23.0

1 0

[PATCH lkp-tests] jobs: change testbox and add queue filed in bisect.yaml
by Cao Xueliang 06 Jan '21

06 Jan '21

testbox: dc-1g-bisect => dc-1g queue: dc-1g-bisect Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 8c2ee8f62..172554a30 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,5 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g-bisect +testbox: dc-1g +queue: dc-1g-bisect bisect: -- 2.23.0

1 0

[PATCH lkp-tests] jobs: dc-1g => dc-1g-bisect in bisect.yaml
by Cao Xueliang 06 Jan '21

06 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 11ffc7c05..8c2ee8f62 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,4 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g +testbox: dc-1g-bisect bisect: -- 2.23.0

2 3

[PATCH compass-ci] lib/job.cr: set docker job os
by Wu Zhende 06 Jan '21

06 Jan '21

docker job's os/os_version/os_mount don't need to define separately. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2554db1..6f344fe 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -153,6 +153,7 @@ class Job private def set_defaults append_init_field() + set_docker_os() set_user_lkp_src() set_os_dir() set_submit_date() @@ -170,6 +171,14 @@ class Job set_subqueue() end + private def set_docker_os + return unless is_docker_job? + + os_info = docker_image.split(":") + self["os"] = os_info[0] + self["os_version"] = os_info[1] + end + private def set_kernel return if os_mount == "container" @@ -309,10 +318,23 @@ class Job @hash[key] = JSON::Any.new(value) if value end + private def is_docker_job? + if testbox =~ /^dc/ + return true + else + return false + end + end + # defaults to the 1st value VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container"] private def set_os_mount + if is_docker_job? + self["os_mount"] = "container" + return + end + if @hash["os_mount"]? if !VALID_OS_MOUNTS.includes?(@hash["os_mount"].to_s) raise "Invalid os_mount: #{@hash["os_mount"]}, should be in #{VALID_OS_MOUNTS}" -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests: add httpress test script
by Wei Jihui 06 Jan '21

06 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/httpress | 136 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100755 tests/httpress diff --git a/tests/httpress b/tests/httpress new file mode 100755 index 00000000..462a2705 --- /dev/null +++ b/tests/httpress @@ -0,0 +1,136 @@ +#!/bin/sh +# - nic +# - nic_ip +# - request_ip +# - request_port +# - request_index +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/env.sh + +: "${request_index:=index.html}" +: "${core_nums:=4}" +: "${protocol:=http}" +: "${alive_type:=short}" + +core_range=0-15 + +http_long_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=50000000 + threads_num=1200 + concurrent_num=1200 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=50000000 + threads_num=400 + concurrent_num=400 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + +http_short_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=400 + concurrent_num=400 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=10000000 + threads_num=300 + concurrent_num=300 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +https_long_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=5000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + +https_short_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=500000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=500000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=1000000 + threads_num=1200 + concurrent_num=1200 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +main() +{ + local request_url="$protocol://$request_ip:$request_port/$request_index" + + ip addr add $nic_ip/24 dev "$nic" + ip link set "$nic" up + + if [ "$protocol" == 'http' ]; then + if [ "$alive_type" == 'long' ]; then + http_long_requests + else + http_short_requests + fi + elif [ "$protocol" == 'https' ]; then + if [ "$alive_type" == 'long' ]; then + https_long_requests + else + https_short_requests + fi + fi +} + +main -- 2.23.0

2 1

[PATCH v5 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 06 Jan '21

06 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 71 ++++++++++++------------------------------------- 1 file changed, 17 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index bf72630..f7ca26e 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,61 +150,24 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� ${CCI_SRC} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ + 1. �� Unable to mount root fs on unknown-block�� - �� ```bash -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: copy rootfs/addon/root/. to container
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 4388e19..b28b24c 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -3,6 +3,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. mkdir -p /root/.ssh -cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ +cp -r /lkp/lkp/src/rootfs/addon/root/. /root cp -r /lkp/lkp/src/rootfs/addon/usr/* /usr sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: copy rootfs/addon/root/. to container
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 +- providers/docker/run.sh | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 4388e19..b28b24c 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -3,6 +3,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. mkdir -p /root/.ssh -cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ +cp -r /lkp/lkp/src/rootfs/addon/root/. /root cp -r /lkp/lkp/src/rootfs/addon/usr/* /usr sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 62a1f9f..57a01c0 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -22,18 +22,19 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --hostname $host -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp - -v ${DIR}/bin:/root/bin:ro + -v ${DIR}/bin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} - /root/bin/entrypoint.sh + /root/sbin/entrypoint.sh ) "${cmd[@]}" -- 2.23.0

1 1