- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 1/2] tests/cci-makepkg: remove useless variable
by Liu Shaofei 06 Jan '21

06 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index d48e8495e..e737955b9 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -108,7 +108,6 @@ distro_install_depends ${benchmark}-dev build_depends_pkg -i $benchmark $TMP sync_dest="$PKG_MNT/$pack_to" -date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: remove useless variable
by Liu Shaofei 06 Jan '21

06 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index d48e8495e..e737955b9 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -108,7 +108,6 @@ distro_install_depends ${benchmark}-dev build_depends_pkg -i $benchmark $TMP sync_dest="$PKG_MNT/$pack_to" -date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -- 2.23.0

1 1

[PATCH lkp-tests 2/2] jobs: add jobs/fault_reproduction.yaml
by Wei Jihui 06 Jan '21

06 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fault_reproduction.yaml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 jobs/fault_reproduction.yaml diff --git a/jobs/fault_reproduction.yaml b/jobs/fault_reproduction.yaml new file mode 100644 index 00000000..46a1e042 --- /dev/null +++ b/jobs/fault_reproduction.yaml @@ -0,0 +1,14 @@ +ssh_pub_key: + <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> +sshd: +sleep: 6h + +on_fail: + sshd: + sleep: 6h -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add new boot way
by Ren Wen 06 Jan '21

06 Jan '21

A new way to find job, requested by: curl -X GET 'http://<host>:<port>/boot.libvirt/mac/<mac_value>' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..f6981e7 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -135,6 +135,29 @@ class Sched return response.to_json end + private def get_boot_libvirt(job : Job) + response = {} of String => String | Nil | Array(String) | Hash(String, String) + + response["job_id"] = job.id + response["kernel_uri"] = job.kernel_uri.lchop("kernel ") + + initrds_uri = job["initrds_uri"]? + initrds_uri = initrds_uri.as_a.map(&.as_s.lchop("initrd ")) if initrds_uri + response["initrds_uri"] = initrds_uri + + kernel_params = job["kernel_params"]? + kernel_params = kernel_params.as_a.map(&.to_s).reject(&.starts_with?("initrd=")).join(" ") if kernel_params + response["kernel_params"] = kernel_params + + response["result_root"] = job.result_root + response["LKP_SERVER"] = "#{job["LKP_SERVER"]?}" + + # Not implement now + response["templates"] = {} of String => String + + return response.to_json + end + private def get_boot_grub(job : Job) initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" @@ -176,6 +199,8 @@ class Sched case boot_type when "ipxe" return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "libvirt" + return job ? get_boot_libvirt(job) : {"info" => "No job now"}.to_json when "grub" return job ? get_boot_grub(job) : grub_msg("No job now") when "container" -- 2.23.0

1 1

[PATCH lkp-tests] lib/detect-system.sh: add system arch riscv64
by Liu Yinsi 06 Jan '21

06 Jan '21

[why] when run job in riscv64 qemu, _system_arch=unknown, it cause qemu quit, so add it. test: /srv/result/iperf/2021-01-06/vm-8p64g/debian-sid-riscv64/tcp-30/z9.281254 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/detect-system.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/detect-system.sh b/lib/detect-system.sh index 90006c8f0..1b344b4f0 100755 --- a/lib/detect-system.sh +++ b/lib/detect-system.sh @@ -31,6 +31,8 @@ parse_executable_arch() _system_arch=aarch64;; *ARM) _system_arch=arm;; + *RISC-V) + _system_arch=riscv64;; *) _system_arch=unknown return 1;; -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: add wget in initramfs-common
by Zhang Yale 06 Jan '21

06 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/initramfs-common | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common index 5d3ecb76..5f766da6 100644 --- a/distro/depends/initramfs-common +++ b/distro/depends/initramfs-common @@ -1,3 +1,4 @@ time +wget perf which -- 2.23.0

1 0

[PATCH lkp-tests] jobs: change testbox and add queue filed in bisect.yaml
by Cao Xueliang 06 Jan '21

06 Jan '21

testbox: dc-1g-bisect => dc-1g queue: dc-1g-bisect Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 8c2ee8f62..172554a30 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,5 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g-bisect +testbox: dc-1g +queue: dc-1g-bisect bisect: -- 2.23.0

1 0

[PATCH lkp-tests] jobs: dc-1g => dc-1g-bisect in bisect.yaml
by Cao Xueliang 06 Jan '21

06 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 11ffc7c05..8c2ee8f62 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,4 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g +testbox: dc-1g-bisect bisect: -- 2.23.0

2 3

[PATCH compass-ci] lib/job.cr: set docker job os
by Wu Zhende 06 Jan '21

06 Jan '21

docker job's os/os_version/os_mount don't need to define separately. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2554db1..6f344fe 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -153,6 +153,7 @@ class Job private def set_defaults append_init_field() + set_docker_os() set_user_lkp_src() set_os_dir() set_submit_date() @@ -170,6 +171,14 @@ class Job set_subqueue() end + private def set_docker_os + return unless is_docker_job? + + os_info = docker_image.split(":") + self["os"] = os_info[0] + self["os_version"] = os_info[1] + end + private def set_kernel return if os_mount == "container" @@ -309,10 +318,23 @@ class Job @hash[key] = JSON::Any.new(value) if value end + private def is_docker_job? + if testbox =~ /^dc/ + return true + else + return false + end + end + # defaults to the 1st value VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container"] private def set_os_mount + if is_docker_job? + self["os_mount"] = "container" + return + end + if @hash["os_mount"]? if !VALID_OS_MOUNTS.includes?(@hash["os_mount"].to_s) raise "Invalid os_mount: #{@hash["os_mount"]}, should be in #{VALID_OS_MOUNTS}" -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests: add httpress test script
by Wei Jihui 06 Jan '21

06 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/httpress | 136 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100755 tests/httpress diff --git a/tests/httpress b/tests/httpress new file mode 100755 index 00000000..462a2705 --- /dev/null +++ b/tests/httpress @@ -0,0 +1,136 @@ +#!/bin/sh +# - nic +# - nic_ip +# - request_ip +# - request_port +# - request_index +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/env.sh + +: "${request_index:=index.html}" +: "${core_nums:=4}" +: "${protocol:=http}" +: "${alive_type:=short}" + +core_range=0-15 + +http_long_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=50000000 + threads_num=1200 + concurrent_num=1200 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=50000000 + threads_num=400 + concurrent_num=400 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + +http_short_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=400 + concurrent_num=400 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=10000000 + threads_num=300 + concurrent_num=300 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +https_long_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=5000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + +https_short_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=500000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=500000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=1000000 + threads_num=1200 + concurrent_num=1200 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +main() +{ + local request_url="$protocol://$request_ip:$request_port/$request_index" + + ip addr add $nic_ip/24 dev "$nic" + ip link set "$nic" up + + if [ "$protocol" == 'http' ]; then + if [ "$alive_type" == 'long' ]; then + http_long_requests + else + http_short_requests + fi + elif [ "$protocol" == 'https' ]; then + if [ "$alive_type" == 'long' ]; then + https_long_requests + else + https_short_requests + fi + fi +} + +main -- 2.23.0

2 1

[PATCH v5 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 06 Jan '21

06 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 71 ++++++++++++------------------------------------- 1 file changed, 17 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index bf72630..f7ca26e 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,61 +150,24 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� ${CCI_SRC} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ + 1. �� Unable to mount root fs on unknown-block�� - �� ```bash -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: copy rootfs/addon/root/. to container
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 4388e19..b28b24c 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -3,6 +3,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. mkdir -p /root/.ssh -cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ +cp -r /lkp/lkp/src/rootfs/addon/root/. /root cp -r /lkp/lkp/src/rootfs/addon/usr/* /usr sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: copy rootfs/addon/root/. to container
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 +- providers/docker/run.sh | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 4388e19..b28b24c 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -3,6 +3,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. mkdir -p /root/.ssh -cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ +cp -r /lkp/lkp/src/rootfs/addon/root/. /root cp -r /lkp/lkp/src/rootfs/addon/usr/* /usr sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 62a1f9f..57a01c0 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -22,18 +22,19 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --hostname $host -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp - -v ${DIR}/bin:/root/bin:ro + -v ${DIR}/bin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} - /root/bin/entrypoint.sh + /root/sbin/entrypoint.sh ) "${cmd[@]}" -- 2.23.0

1 1

[PATCH compass-ci] mail: rename Compass-ci to Compass-CI
by Wu Fengguang 06 Jan '21

06 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/mail_job_result.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 50e2f53..796a4f3 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -22,8 +22,8 @@ class MailJobResult def compose_mail set_submitter_info - subject = "[Compass-ci] job: #{@job_id} result" - signature = "Regards\nCompass-ci\nhttps://gitee.com/openeuler/compass-ci" + subject = "[Compass-CI] job: #{@job_id} result" + signature = "Regards\nCompass-CI\nhttps://gitee.com/openeuler/compass-ci" body = "Hi, Thanks for your participation in Kunpeng and software ecosystem! Your Job: #{@job_id} had finished. -- 2.23.0

1 0

[PATCH compass-ci] sparrow: add setup.yaml
by Liu Yinsi 06 Jan '21

06 Jan '21

[why] before deploy compass-ci, fill configuration setup.yaml firstly, will make the whole process cleaner. fill my_name and my_email, it's necessary to verify account when submit job. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/setup.yaml | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 sparrow/setup.yaml diff --git a/sparrow/setup.yaml b/sparrow/setup.yaml new file mode 100644 index 0000000..b1cac8c --- /dev/null +++ b/sparrow/setup.yaml @@ -0,0 +1,11 @@ +# please copy this template with the below commands, +# then fill the configuration according to comments. +# mkdir -p /etc/compass-ci/ +# cp setup.yaml /etc/compass-ci/setup.yaml +# vim /etc/compass-ci/setup.yaml + +# account name +my_name: xxxxxx + +# email address +my_email: xxx@xxx -- 2.23.0

3 4

[PATCH compass-ci] doc: update doc
by Luan Shengde 06 Jan '21

06 Jan '21

rename email address compass-ci-reply(a)qq.com => compass-ci-robot(a)qq.com Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 2 +- doc/manual/apply-account.md | 2 +- doc/manual/borrow-machine.en.md | 2 +- doc/manual/borrow-machine.zh.md | 2 +- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index e97912c..6158b65 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -11,7 +11,7 @@ apply account/uuid for user ## usage build email as the following section: allowed email format -send mail to: compass-ci-reply(a)qq.com +send mail to: compass-ci-robot(a)qq.com ## allowed email format: diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 08e7618..a0b2c82 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci-reply(a)qq.com + To: compass-ci-robot(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..1b4a220 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -3,7 +3,7 @@ Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - Send an email to apply for an account. -- Receive an email from compass-ci-reply(a)qq.com. +- Receive an email from compass-ci-robot(a)qq.com. - Configure the local environment. # Applying for a Test Machine (VM) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..bb1c08f 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -2,7 +2,7 @@ 请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - send apply account email. -- receive email from compass-ci-reply(a)qq.com. +- receive email from compass-ci-robot(a)qq.com. - local environment configuration. diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 6f0736b..ad933b1 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -6,7 +6,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email -- receive email from compass-ci-reply(a)qq.com +- receive email from compass-ci-robot(a)qq.com - get reply email and follow its instructions to - setup default config -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: change the mount path
by Xiao Shenwei 06 Jan '21

06 Jan '21

[why] /root/bin the dir may be used by user to stores own file --- providers/docker/run.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 62a1f9f..57a01c0 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -22,18 +22,19 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --hostname $host -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp - -v ${DIR}/bin:/root/bin:ro + -v ${DIR}/bin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} - /root/bin/entrypoint.sh + /root/sbin/entrypoint.sh ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/build-my-info.rb: modify method of inputting variables
by Hu Xuejiao 06 Jan '21

06 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/build-my-info.rb | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 40193af..2562062 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -3,16 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -require 'io/console' require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' +require 'optparse' -print 'email: ' -my_email = $stdin.echo = gets.chomp -print 'name: ' -my_name = $stdin.echo = gets.chomp -print 'lab: ' -lab = $stdin.echo = gets.chomp +option = {} +options= OptionParser.new do |opts| + opts.on('-e email', 'my_email') do |email| + option[:email] = email + end -build_my_info = BuildMyInfo.new(my_email, my_name, lab) + opts.on('-n name', 'my_name') do |name| + option[:name] = name + end + + opts.on('-l lab', 'lab') do |lab| + option[:lab] = lab + end +end + +options.parse! + +build_my_info = BuildMyInfo.new(option[:my_email],option[:my_name],option[:lab]) build_my_info.config_my_info -- 2.23.0

3 3

[PATCH lkp-tests] sshd: improve notification email
by Wu Fengguang 06 Jan '21

06 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- daemon/sshd | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index c2404dbfd..e9f380060 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -39,13 +39,11 @@ run_ssh() data_success() { data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application successful +Subject: [NOTIFY Compass-CI] Machine application successful -Dear $my_username: +Dear $my_username, - Thanks for your participation in software ecosystem! - According to your application, the test machine has been provisioned. - The datails are as follows: + Your test machine has been provisioned. Login: ssh root(a)api.compass-ci.openeuler.org -p $port @@ -59,7 +57,7 @@ Dear $my_username: $os $os_version $os_arch Regards -Compass-Ci +Compass-CI " } -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add httpress test script
by Wei Jihui 06 Jan '21

06 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/httpress | 138 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100755 tests/httpress diff --git a/tests/httpress b/tests/httpress new file mode 100755 index 00000000..f8b0e4d5 --- /dev/null +++ b/tests/httpress @@ -0,0 +1,138 @@ +#!/bin/sh +# - nic +# - nic_ip +# - request_ip +# - request_port +# - request_index +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/env.sh + +: "${request_index:=index.html}" +: "${core_nums:=4}" +: "${protocol:=http}" +: "${alive_type:=short}" + + +core_range=0-15 + +http_long_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=50000000 + threads_num=1200 + concurrent_num=1200 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=50000000 + threads_num=400 + concurrent_num=400 + ethtool -L $nic combined 16 && /lkp/lkp/src/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + + +http_short_requests() +{ + if [ "$core_nums" == 4 ]; then + requests_num=3000000 + threads_num=300 + concurrent_num=300 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=400 + concurrent_num=400 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=10000000 + threads_num=300 + concurrent_num=300 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + ethtool -L $nic combined 16 && $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + numactl -C 16-95 httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +https_long_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=5000000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=5000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=10000000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" -k "$request_url" + fi +} + +https_short_requests() +{ + if [ "$core_nums" == 4 ] || [ "$core_nums" == 8 ]; then + requests_num=500000 + threads_num=100 + concurrent_num=100 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 48 ]; then + requests_num=500000 + threads_num=600 + concurrent_num=600 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + elif [ "$core_nums" == 96 ]; then + requests_num=1000000 + threads_num=1200 + concurrent_num=1200 + httpress -n "$requests_num" -c "$concurrent_num" -t "$threads_num" "$request_url" + fi +} + +main() +{ + local request_url="$protocol://$request_ip:$request_port/$request_index" + + ip addr add $nic_ip/24 dev "$nic" + ip link set "$nic" up + + if [ "$protocol" == 'http' ]; then + if [ "$alive_type" == 'long' ]; then + http_long_requests + else + http_short_requests + fi + elif [ "$protocol" == 'https' ]; then + if [ "$alive_type" == 'long' ]; then + https_long_requests + else + https_short_requests + fi + fi +} + +main -- 2.23.0

4 3

[PATCH v3 compass-ci] sbin/build-my-info.rb: modify method of inputting variables
by Hu Xuejiao 06 Jan '21

06 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/build-my-info.rb | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 40193af..cd824c6 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -3,16 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -require 'io/console' require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' +require 'optparse' -print 'email: ' -my_email = $stdin.echo = gets.chomp -print 'name: ' -my_name = $stdin.echo = gets.chomp -print 'lab: ' -lab = $stdin.echo = gets.chomp +option = {} +options= OptionParser.new do |opts| + opts.on('-e email', 'my_email') do |email| + option[:email] = email + end -build_my_info = BuildMyInfo.new(my_email, my_name, lab) + opts.on('-n name', 'my_name') do |name| + option[:name] = name + end + + opts.on('-l lab', 'lab') do |lab| + option[:lab] = lab + end +end + +options.parse! + +build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info -- 2.23.0

1 0

[PATCH lkp-tests] installer/debian: fix the exit problem when source this file
by Yu Chuan 06 Jan '21

06 Jan '21

[Why] The purpose of adding the last four lines is to remove useless warn logs, but it exits at the end. This will cause the process which source this script exit. [Addition] Related commit id: 16cef539c517d0e6bd3495a1e3079214a8b4bd24 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/installer/debian | 4 ---- 1 file changed, 4 deletions(-) diff --git a/distro/installer/debian b/distro/installer/debian index 3148ee3b2c2a..35b3209bd747 100755 --- a/distro/installer/debian +++ b/distro/installer/debian @@ -6,7 +6,3 @@ dpkg --add-architecture i386 && apt-get update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ -y install $* >/tmp/apt-get_info 2>&1 - -ret_value=$? -grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info -exit $ret_value -- 2.23.0

1 0

[PATCH compass-ci 2/3] providers/docker: change the dir name bin to sbin
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/{ => sbin}/bin/entrypoint.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename providers/docker/{ => sbin}/bin/entrypoint.sh (100%) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/sbin/bin/entrypoint.sh similarity index 100% rename from providers/docker/bin/entrypoint.sh rename to providers/docker/sbin/bin/entrypoint.sh -- 2.23.0

1 1

[PATCH compass-ci] addsoftware: clean the runtime dir after modifying nfsroot
by Xu Xijian 06 Jan '21

06 Jan '21

When install a software into a nfsroot, some unwanted files will be added into runtime dir, so rm them to keep nfsroot clean. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 62d4eff..60e5e07 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -40,7 +40,11 @@ install_cmds=( --rm -v "$rootdir":/mnt/root addsoftware:auto - yum install -y --skip-broken --installroot=/mnt/root $packages + bash -c " + yum install -y --skip-broken --installroot=/mnt/root $packages + cd /mnt/root + rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + " ) echo "rsync rootfs from $basever to $newver..." -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/docker: copy rootfs/addon/root/* to container
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/sbin/entrypoint.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 providers/docker/sbin/entrypoint.sh diff --git a/providers/docker/sbin/entrypoint.sh b/providers/docker/sbin/entrypoint.sh new file mode 100755 index 0000000..9848b4f --- /dev/null +++ b/providers/docker/sbin/entrypoint.sh @@ -0,0 +1,7 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +cp -r /lkp/lkp/src/rootfs/addon/root/. /root +cp -r /lkp/lkp/src/rootfs/addon/usr/* /usr +sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci 1/3] providers/docker: change the file path of entrypoint.sh
by Xiao Shenwei 06 Jan '21

06 Jan '21

[why] /root/bin the dir may be used by user to stores own file Thanks, Shenwei --- providers/docker/run.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 62a1f9f..d39b6cf 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,17 +23,18 @@ cmd=( docker run --rm -m $memory + --hostname $host --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp - -v ${DIR}/bin:/root/bin:ro + -v ${DIR}/sbin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} - /root/bin/entrypoint.sh + /root/sbin/entrypoint.sh ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix check key
by Wei Jihui 06 Jan '21

06 Jan '21

first check key starts with "#!", then check key matchs /^\w.*\|.*\w$/ Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 35e0ceee..8346db87 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -288,7 +288,6 @@ class Job def multi_args(hash) jobs_array = [] hash.each { |key, value| - next unless key =~ /^\w.*\|.*\w$/ next unless key.is_a?(String) if key.start_with?('#! ') @@ -296,6 +295,8 @@ class Job next end + next unless key =~ /^\w.*\|.*\w$/ + key_array = key.split('|').map(&:strip) [value].flatten.each do |v| next unless v =~ /^\w.*\|.*\w$/ -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that project name start with capital letter can't match
by Li Yuanchao 06 Jan '21

06 Jan '21

Some repositories' name start with capital letter, but upstream-repos all start with lower case letter. So such repository can't match git_repo when use webhook container. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index ce1dc9e..bd55b33 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -333,7 +333,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" return git_repo if check_git_repo(git_repo, webhook_url) - git_repo = "#{project[0]}/#{project}/#{project}" + git_repo = "#{project[0].downcase}/#{project}/#{project}" return git_repo if check_git_repo(git_repo, webhook_url) puts "webhook: #{webhook_url} is not found!" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: make gem install system wide
by Wu Fengguang 06 Jan '21

06 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sparrow/0-package/common | 2 +- sparrow/5-build/overcommit | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 57ab701..4a9da60 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/5-build/overcommit b/sparrow/5-build/overcommit index e8add6f..af02d84 100755 --- a/sparrow/5-build/overcommit +++ b/sparrow/5-build/overcommit @@ -10,4 +10,4 @@ cd overcommit oc_build_out=$(gem build overcommit.gemspec | grep "overcommit-.*\.gem") -gem install "$oc_build_out" +gem install --no-user-install "$oc_build_out" -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: fix rubocop warning
by Li Yuanchao 06 Jan '21

06 Jan '21

git_mirror.rb:352:35: W: Lint/UnusedMethodArgument: Unused method argument - parent_project. If it's necessary, use _ or _parent_project as an argument name to indicate that it won't be used. def handle_submodule(submodule, parent_project) ^^^^^^^^^^^^^^ git_mirror.rb:358:7: W: Lint/NonLocalExitFromIterator: Non-local exit from iterator, without return value. next, break, Array#find, Array#any?, etc. is preferred. return unless git_repo ^^^^^^ Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f265d9..ce1dc9e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -349,13 +349,13 @@ class MirrorMain end end - def handle_submodule(submodule, parent_project) + def handle_submodule(submodule) submodule.each_line do |line| next unless line.include?('url = ') url = line.split(' = ')[1].chomp git_repo = url.split('://')[1] if url.include?('://') - return unless git_repo + break unless git_repo @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } fork_stat_init(git_repo) -- 2.23.0

1 0

[PATCH lkp-tests] tests/host-info: computing vm memory
by Wei Jihui 06 Jan '21

06 Jan '21

[why] it can not compute vm memory when vm memory less than 16G. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/host-info | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index fe55d1ff..2721648c 100755 --- a/tests/host-info +++ b/tests/host-info @@ -4,8 +4,8 @@ get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') - memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G echo "memory: $memory" } -- 2.23.0

3 2

[PATCH v4 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 06 Jan '21

06 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 71 ++++++++++++------------------------------------- 1 file changed, 17 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index bf72630..bab40bc 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,61 +150,24 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ + 1. �� Unable to mount root fs on unknown-block�� - �� ```bash -- 2.23.0

2 3

[PATCH lkp-tests 1/2] stats/crystal/kernbench.cr: create kernbench.cr as same content with stats/kernbench
by Lu Kaiyi 06 Jan '21

06 Jan '21

create kernbench.cr in crystal directory from stats/kernbench first, and then, the second patch will know what has changed from original file. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 stats/crystal/kernbench.cr diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr new file mode 100644 index 000000000..0da37fe88 --- /dev/null +++ b/stats/crystal/kernbench.cr @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +$stdin.each_line do |line| + case line + when /Elapsed Time/ + puts 'elapsed_time: ' + line.split[2] + puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /User Time/ + puts 'user_time: ' + line.split[2] + puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /System Time/ + puts 'system_time: ' + line.split[2] + puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Percent CPU/ + puts 'percent_cpu: ' + line.split[2] + puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Context Switches/ + puts 'context_switches: ' + line.split[2] + puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Sleeps/ + puts 'sleeps: ' + line.split[1] + puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + end +end -- 2.23.0

2 1

[PATCH compass-ci] git_mirror.rb: optimize method to pass rubocop
by Li Yuanchao 06 Jan '21

06 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f265d9..ce1dc9e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -349,13 +349,13 @@ class MirrorMain end end - def handle_submodule(submodule, parent_project) + def handle_submodule(submodule) submodule.each_line do |line| next unless line.include?('url = ') url = line.split(' = ')[1].chomp git_repo = url.split('://')[1] if url.include?('://') - return unless git_repo + break unless git_repo @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } fork_stat_init(git_repo) -- 2.23.0

2 1

[PATCH lab-z9] hosts: add new testbox host info
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/taishan200-2280-2s64p-256g--a115 | 20 ++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a117 | 20 ++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a119 | 20 ++++++++++++++++++++ hosts/vm-2p32g | 6 ++++++ 4 files changed, 66 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a115 create mode 100644 hosts/taishan200-2280-2s64p-256g--a117 create mode 100644 hosts/taishan200-2280-2s64p-256g--a119 create mode 100644 hosts/vm-2p32g diff --git a/hosts/taishan200-2280-2s64p-256g--a115 b/hosts/taishan200-2280-2s64p-256g--a115 new file mode 100644 index 0000000..64e2b1a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a115 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399888177f5 +rootfs_partition: +- /dev/disk/by-id/scsi-35000039988817359 +mac_addr: + - 44:a1:91:9d:f3:44 + - 44:a1:91:9d:f3:45 + - 44:a1:91:9d:f3:46 + - 44:a1:91:9d:f3:47 + - 44:a1:91:e5:94:51 + - 44:a1:91:e5:94:52 + - 44:a1:91:e5:94:53 + - 44:a1:91:e5:94:54 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.8 diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 new file mode 100644 index 0000000..31167f9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b57d1 +rootfs_partition: +- /dev/disk/by-id/scsi-350000399886b720d +mac_addr: + - 44:67:47:85:f4:0b + - 44:67:47:85:f4:0c + - 44:67:47:85:f4:0d + - 44:67:47:85:f4:0e + - cc:05:77:ee:b3:ff + - cc:05:77:ee:b4:00 + - cc:05:77:ee:b4:01 + - cc:05:77:ee:b4:02 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 new file mode 100644 index 0000000..5775be3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b5dbd +rootfs_partition: +- /dev/disk/by-id/scsi-350000399886b6be9 +mac_addr: + - 44:67:47:e9:79:65 + - 44:67:47:e9:79:66 + - 44:67:47:e9:79:67 + - 44:67:47:e9:79:68 + - 44:a1:91:9f:9b:4a + - 44:a1:91:9f:9b:4b + - 44:a1:91:9f:9b:4c + - 44:a1:91:9f:9b:4d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.12 diff --git a/hosts/vm-2p32g b/hosts/vm-2p32g new file mode 100644 index 0000000..fcb23ed --- /dev/null +++ b/hosts/vm-2p32g @@ -0,0 +1,6 @@ +arch: aarch64 +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 2/2] stats/crystal/kernbench.cr: kernbench to kernbench.cr
by Lu Kaiyi 06 Jan '21

06 Jan '21

change kernbench into kernbench.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr index 0da37fe88..9777a11ca 100644 --- a/stats/crystal/kernbench.cr +++ b/stats/crystal/kernbench.cr @@ -1,28 +1,26 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal -LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) +require "../../lib/statistics" -require "#{LKP_SRC}/lib/statistics" - -$stdin.each_line do |line| +STDIN.each_line do |line| case line when /Elapsed Time/ - puts 'elapsed_time: ' + line.split[2] - puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "elapsed_time: " + line.split[2] + puts "elapsed_time_stddev: " + line.split[3].gsub(/[()]/, "") when /User Time/ - puts 'user_time: ' + line.split[2] - puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "user_time: " + line.split[2] + puts "user_time_stddev: " + line.split[3].gsub(/[()]/, "") when /System Time/ - puts 'system_time: ' + line.split[2] - puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "system_time: " + line.split[2] + puts "system_time_stddev: " + line.split[3].gsub(/[()]/, "") when /Percent CPU/ - puts 'percent_cpu: ' + line.split[2] - puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "percent_cpu: " + line.split[2] + puts "percent_cpu_stddev: " + line.split[3].gsub(/[()]/, "") when /Context Switches/ - puts 'context_switches: ' + line.split[2] - puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "context_switches: " + line.split[2] + puts "context_switches_stddev: " + line.split[3].gsub(/[()]/, "") when /Sleeps/ - puts 'sleeps: ' + line.split[1] - puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + puts "sleeps: " + line.split[1] + puts "sleeps_stddev: " + line.split[2].gsub(/[()]/, "") end end -- 2.23.0

1 0

[PATCH v3 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 06 Jan '21

06 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v3 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 06 Jan '21

06 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v3 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 06 Jan '21

06 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..a5ca748 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 06 Jan '21

06 Jan '21

example: name => email_addr email_addr_a => email_addr email_addr_b => email_addr an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..f190896 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 06 Jan '21

06 Jan '21

timing work: reset email rate limit queues everyday Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..099d89d --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to enable to send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 06 Jan '21

06 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v3 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 06 Jan '21

06 Jan '21

[why] the service need to install redis to store email counts to redis queue the service need to install rufus-scheduler to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH compass-ci 2/2] compare: remove not latest failed metrics
by Lu Weitao 06 Jan '21

06 Jan '21

[Why] User just need the failed metrics belong latest job in compare result [How] 1. query job_list and order by start_time desc 2. auto group job_list - the first job of each dimension is the latest job { group_key => { dim1 => [job1, ...] ... } } 3. get latest job_list for each group 4. in compare result, ignore the failed metric that not belongs the latest job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 32 ++++++++++++++++++++++++-------- lib/matrix2.rb | 24 +++++++++++++++++------- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 6a2f515..21e24df 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_hash, options) + groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 21bd5c9..7747ea1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -159,7 +159,7 @@ def get_matrixes_fields(matrixes_list) matrixes_fields end -def get_matrixes_values(matrixes_list, options) +def get_matrixes_values(matrixes_list, latest_jobs, options) # get all matrixes all field values # matrixes_values = { false => {}, true => {} } @@ -168,6 +168,10 @@ def get_matrixes_values(matrixes_list, options) next if field == 'stats_source' success = successful?(field) + unless success + next unless latest_failure?(field, latest_jobs) + end + matrixes_values[success][field] = get_values_by_field( matrixes_list, field, matrixes_size, success, options @@ -176,6 +180,18 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end +def latest_failure?(field, latest_jobs) + flag = false + latest_jobs.each do |job| + if job['stats'][field] + flag = true + break + end + end + + flag +end + def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # @@ -197,7 +213,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -206,7 +222,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key return warn('Matrix cannot be empty!') || '' if matrixes_empty?(matrixes_list) options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) - matrixes_values = get_matrixes_values(matrixes_list, options) + matrixes_values = get_matrixes_values(matrixes_list, latest_jobs, options) remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' @@ -387,28 +403,28 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_hash, options) +def compare_group_matrices(group_matrices, suites_hash, latest_jobs_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| next if have_multi_member && matrice_kv[1].size < 2 - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], latest_jobs_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suite_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, latest_jobs, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 09573ec..e3ae580 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -115,27 +115,37 @@ end # } def combine_group_query_data(job_list, dims) suites_hash = {} + latest_jobs_hash = {} groups = auto_group(job_list, dims) have_multi_member = multi_member?(groups) one_size_count = 0 groups.each do |group_key, value| if value.size < 2 one_size_count += 1 - if have_multi_member || one_size_count > 3 - groups.delete(group_key) - next - end + next if delete_group?(groups, group_key, have_multi_member, one_size_count) end - get_groups_matrix(groups, group_key, value, suites_hash) + get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end - return groups, suites_hash + return groups, suites_hash, latest_jobs_hash end -def get_groups_matrix(groups, group_key, value, suites_hash) +def delete_group?(groups, group_key, have_multi_member, one_size_count) + was_deleted = false + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + was_deleted = true + end + + was_deleted +end + +def get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) suite_list = [] + latest_jobs_hash[group_key] = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) + latest_jobs_hash[group_key] << jobs[0] end suites_hash[group_key] = suite_list end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare.rb: order query result with desc
by Lu Weitao 06 Jan '21

06 Jan '21

[Why] 1. When query result include more than 10,000 jobs, we still need the latest jobs use to compare 2. Support we recognize the latest job that have failure messages from each group-dimension-job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 9090c3b..6a2f515 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -40,7 +40,7 @@ def create_matrices_list(conditions) suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| - query_results = es.multi_field_query(condition) + query_results = es.multi_field_query(condition, desc_keyword: 'start_time') matrix, suites = combine_query_data(query_results) matrices_list << matrix suite_list.concat(suites) @@ -68,7 +68,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) - query_results = es.multi_field_query(conditions) + query_results = es.multi_field_query(conditions, desc_keyword: 'start_time') combine_group_query_data(query_results['hits']['hits'], dims) end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: git clone only one commit at first
by Li Yuanchao 05 Jan '21

05 Jan '21

Some repositories have many commits which make it big, and cause overtime when git clone. Clone only one commit at first, and pull all commits later, can help clone large repositories. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 156d5bb..7f265d9 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -42,7 +42,7 @@ class GitMirror ret = false url = get_url(Array(url)[0]) 10.times do - stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) ret = !stderr.include?('fatal') break if ret end @@ -51,6 +51,11 @@ class GitMirror end def git_fetch(mirror_dir) + if File.exist?("#{mirror_dir}/shallow") + %x(git -C #{mirror_dir} fetch --unshallow 2>&1) + return false + end + fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) # Check whether mirror_dir is a good git repository by 2 conditions. If not, delete it. if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/run_job: registering user when submit-job
by Hu Xuejiao 05 Jan '21

05 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/run_job b/sbin/run_job index c84df0d..325fa9f 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -1,5 +1,8 @@ #!/bin/bash +# register user information +ruby build-my-info.rb -e "$my_email" -n "$USER" -l "$lab" + cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file -- 2.23.0

1 0

[PATCH v4 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 05 Jan '21

05 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e7b5084 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # Please ensure $HOME/.config/compass-ci/rootfs-passwd exists. + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs/run: fix the incorrect image names
by Wang Chenglong 05 Jan '21

05 Jan '21

[Why] When docker's image name contains multilevel addresses, it needs to intercept again. image name: docker.io/openeuler20.03-LTS package name: openeuler20.03-LTS.img Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run index cf91337..ce51ba3 100755 --- a/container/docker-rootfs/run +++ b/container/docker-rootfs/run @@ -11,7 +11,8 @@ check_cmd_input "$@" check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag -IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) +IMAGE_NAME=${docker_name_tag%%:*} +IMAGE_PACK=${IMAGE_NAME##*/}.cgz start=( docker run -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/run: fix the incorrect image names
by Wang Chenglong 05 Jan '21

05 Jan '21

[Why] When docker's image name contains multilevel addresses, it needs to intercept again. image name: docker.io/openeuler20.03-LTS package name: openeuler20.03-LTS.img Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run index cf91337..8a7b315 100755 --- a/container/docker-rootfs/run +++ b/container/docker-rootfs/run @@ -11,7 +11,8 @@ check_cmd_input "$@" check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag -IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) +IMAGE_NAME=${docker_name_tag%%:*} +IMAGE_PACK=$(echo ${IMAGE_NAME##*/}.cgz) start=( docker run -- 2.23.0

2 2