October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] [why] current container will only bind to special ip (172.17.0.1 and the configured TASKQUEUE_HOST at default configure file). some servier default visit 172.17.0.1, but some visit (at crystal) 172.168.131.113 as default.
by Tong Qunfeng 27 Oct '20

27 Oct '20

[how] 1. add TASKQUEUE_HOST to system default configure file (/etc/compass-ci/defaults/*.yaml) 2. add TASKQUEUE_HOST to taskqueue container start file. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- container/taskqueue/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/start b/container/taskqueue/start index bdd5e7c..45a8b03 100755 --- a/container/taskqueue/start +++ b/container/taskqueue/start @@ -8,6 +8,7 @@ require_relative '../defconfig.rb' names = Set.new %w[ TASKQUEUE_PORT + TASKQUEUE_HOST REDIS_HOST REDIS_PORT ] -- 2.23.0

1 1

[PATCH compass-ci] fix: taskqueue: add TQSKQUEUE_HOST to container
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] current container will only bind to special ip (172.17.0.1 and the configured TASKQUEUE_HOST at default configure file). some servier default visit 172.17.0.1, but some visit (at crystal) 172.168.131.113 as default. [how] 1. add TASKQUEUE_HOST to system default configure file (/etc/compass-ci/defaults/*.yaml) 2. add TASKQUEUE_HOST to taskqueue container start file. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- container/taskqueue/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/start b/container/taskqueue/start index bdd5e7c..45a8b03 100755 --- a/container/taskqueue/start +++ b/container/taskqueue/start @@ -8,6 +8,7 @@ require_relative '../defconfig.rb' names = Set.new %w[ TASKQUEUE_PORT + TASKQUEUE_HOST REDIS_HOST REDIS_PORT ] -- 2.23.0

1 1

[PATCH v4 compass-ci] fix: taskqueue: add TASKQUEUE_HOST to container
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] current container will only bind to special ip (172.17.0.1 and the configured TASKQUEUE_HOST at default configure file). some servier default visit 172.17.0.1, but some visit (at crystal) 172.168.131.113 as default. [how] 1. add TASKQUEUE_HOST to system default configure file (/etc/compass-ci/defaults/*.yaml) 2. add TASKQUEUE_HOST to taskqueue container start file. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- container/taskqueue/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/start b/container/taskqueue/start index bdd5e7c..45a8b03 100755 --- a/container/taskqueue/start +++ b/container/taskqueue/start @@ -8,6 +8,7 @@ require_relative '../defconfig.rb' names = Set.new %w[ TASKQUEUE_PORT + TASKQUEUE_HOST REDIS_HOST REDIS_PORT ] -- 2.23.0

1 0

[PATCH lab-z9 1/3] new hosts file
by Zhang Dewan 27 Oct '20

27 Oct '20

--- hosts/m | 40 ++++++++++++++++++++++++++++++++++++++++ hosts/p | 10 ++++++++++ 2 files changed, 50 insertions(+) create mode 100644 hosts/m create mode 100644 hosts/p diff --git a/hosts/m b/hosts/m new file mode 100644 index 0000000..b73dfb7 --- /dev/null +++ b/hosts/m @@ -0,0 +1,40 @@ +taishan200-2280-2s48p-256g--a11 a11 +taishan200-2280-2s48p-256g--a13 a13 +taishan200-2280-2s48p-256g--a14 a14 +taishan200-2280-2s48p-256g--a15 a15 +taishan200-2280-2s48p-256g--a17 a17 +taishan200-2280-2s48p-256g--a18 a18 +taishan200-2280-2s48p-256g--a19 a19 +taishan200-2280-2s48p-256g--a21 a21 +taishan200-2280-2s48p-256g--a22 a22 +taishan200-2280-2s48p-256g--a24 a24 +taishan200-2280-2s48p-256g--a25 a25 +taishan200-2280-2s48p-256g--a26 a26 +taishan200-2280-2s48p-256g--a27 a27 +taishan200-2280-2s48p-256g--a28 a28 +taishan200-2280-2s48p-256g--a29 a29 +taishan200-2280-2s48p-256g--a30 a30 +taishan200-2280-2s48p-256g--a31 a31 +taishan200-2280-2s48p-256g--a32 a32 +taishan200-2280-2s48p-256g--a33 a33 +taishan200-2280-2s48p-256g--a34 a34 +taishan200-2280-2s48p-256g--a52 a52 +taishan200-2280-2s48p-256g--a53 a53 +taishan200-2280-2s48p-256g--a60 a60 +taishan200-2280-2s48p-256g--a61 a61 +taishan200-2280-2s48p-256g--a62 a62 +taishan200-2280-2s48p-256g--a66 a66 +taishan200-2280-2s48p-256g--a67 a67 +taishan200-2280-2s48p-256g--a68 a68 +taishan200-2280-2s48p-256g--a69 a69 +taishan200-2280-2s48p-256g--a70 a70 +taishan200-2280-2s48p-256g--a71 a71 +taishan200-2280-2s64p-256g--a10 a10 +taishan200-2280-2s64p-256g--a3 a3 +taishan200-2280-2s64p-256g--a36 a36 +taishan200-2280-2s64p-256g--a4 a4 +taishan200-2280-2s64p-256g--a40 a40 +taishan200-2280-2s64p-256g--a41 a41 +taishan200-2280-2s64p-256g--a42 a42 +taishan200-2280-2s64p-256g--a8 a8 +taishan200-2280-2s64p-256g--a9 a9 diff --git a/hosts/p b/hosts/p new file mode 100644 index 0000000..9d47282 --- /dev/null +++ b/hosts/p @@ -0,0 +1,10 @@ +#!/bin/bash + +while read line +do + + i=$(echo $line | awk '{print $2}') + j=$(echo $line | awk '{print $1}') + ip=$(grep $i /home/dewan/servers.info | head -n 1 |awk '{print $1}') + echo "ipmi_ip: $ip" >> $j +done < m -- 2.23.0

5 5

[PATCH compass-ci] sparrow/0-package: add epel-release package.
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] Before install redis and lockfile-progs in centos, need install epel-release package first. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- sparrow/0-package/os/centos | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 261592c..8dcab75 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -3,6 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. pkgs=( + epel-release vim git gcc -- 2.23.0

2 1

[PATCH compass-ci] sparrow/0-package: add restart redis
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] When redis was installed in centos, It need to start redis.service. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 31588f4..f73f2c4 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -40,3 +40,4 @@ cat > /etc/docker/daemon.json <<EOF EOF systemctl restart docker +systemctl restart redis -- 2.23.0

1 0

[PATCH lab-z9 1/2] new hosts file
by Zhang Dewan 27 Oct '20

27 Oct '20

--- hosts/m | 40 ++++++++++++++++++++++++++++++++++++++++ hosts/p | 10 ++++++++++ 2 files changed, 50 insertions(+) create mode 100644 hosts/m create mode 100644 hosts/p diff --git a/hosts/m b/hosts/m new file mode 100644 index 0000000..b73dfb7 --- /dev/null +++ b/hosts/m @@ -0,0 +1,40 @@ +taishan200-2280-2s48p-256g--a11 a11 +taishan200-2280-2s48p-256g--a13 a13 +taishan200-2280-2s48p-256g--a14 a14 +taishan200-2280-2s48p-256g--a15 a15 +taishan200-2280-2s48p-256g--a17 a17 +taishan200-2280-2s48p-256g--a18 a18 +taishan200-2280-2s48p-256g--a19 a19 +taishan200-2280-2s48p-256g--a21 a21 +taishan200-2280-2s48p-256g--a22 a22 +taishan200-2280-2s48p-256g--a24 a24 +taishan200-2280-2s48p-256g--a25 a25 +taishan200-2280-2s48p-256g--a26 a26 +taishan200-2280-2s48p-256g--a27 a27 +taishan200-2280-2s48p-256g--a28 a28 +taishan200-2280-2s48p-256g--a29 a29 +taishan200-2280-2s48p-256g--a30 a30 +taishan200-2280-2s48p-256g--a31 a31 +taishan200-2280-2s48p-256g--a32 a32 +taishan200-2280-2s48p-256g--a33 a33 +taishan200-2280-2s48p-256g--a34 a34 +taishan200-2280-2s48p-256g--a52 a52 +taishan200-2280-2s48p-256g--a53 a53 +taishan200-2280-2s48p-256g--a60 a60 +taishan200-2280-2s48p-256g--a61 a61 +taishan200-2280-2s48p-256g--a62 a62 +taishan200-2280-2s48p-256g--a66 a66 +taishan200-2280-2s48p-256g--a67 a67 +taishan200-2280-2s48p-256g--a68 a68 +taishan200-2280-2s48p-256g--a69 a69 +taishan200-2280-2s48p-256g--a70 a70 +taishan200-2280-2s48p-256g--a71 a71 +taishan200-2280-2s64p-256g--a10 a10 +taishan200-2280-2s64p-256g--a3 a3 +taishan200-2280-2s64p-256g--a36 a36 +taishan200-2280-2s64p-256g--a4 a4 +taishan200-2280-2s64p-256g--a40 a40 +taishan200-2280-2s64p-256g--a41 a41 +taishan200-2280-2s64p-256g--a42 a42 +taishan200-2280-2s64p-256g--a8 a8 +taishan200-2280-2s64p-256g--a9 a9 diff --git a/hosts/p b/hosts/p new file mode 100644 index 0000000..9d47282 --- /dev/null +++ b/hosts/p @@ -0,0 +1,10 @@ +#!/bin/bash + +while read line +do + + i=$(echo $line | awk '{print $2}') + j=$(echo $line | awk '{print $1}') + ip=$(grep $i /home/dewan/servers.info | head -n 1 |awk '{print $1}') + echo "ipmi_ip: $ip" >> $j +done < m -- 2.23.0

4 3

[PATCH compass-ci 3/5] container/send-mail: Dockerfile
by Luan Shengde 27 Oct '20

27 Oct '20

updata Dockerfile Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..d151bda 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,18 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive -COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone - -EXPOSE 25 - -COPY mail-post.rb /usr/local/bin/ -COPY send-mail.rb /usr/local/bin/ - -CMD ["/usr/local/bin/mail-post.rb"] + apt-get install -y ruby-mail ruby-json ruby-sinatra -- 2.23.0

2 1

[PATCH compass-ci 4/5] container/send-mail: start
by Luan Shengde 27 Oct '20

27 Oct '20

update start Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 68 +++++++++++++-------------------------- 1 file changed, 22 insertions(+), 46 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..95d6b02 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,60 +1,36 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( +names = Set.new %w[ + LOCAL_SEND_MAIL_PORT + LOCAL_ROBOT_EMAIL_ADDRESS +] + +defaults = relevant_defaults(names) + +cmd = %W[ docker run --restart=always --name=send-mail -u nobody -d - -p 11311:11311 + -e CCI_SRC=/c/compass-ci + -e SEND_MAIL_PORT=#{defaults['LOCAL_SEND_MAIL_PORT']} + -e ROBOT_EMAIL_ADDRESS=#{defaults['LOCAL_ROBOT_EMAIL_ADDRESS']} + -p #{defaults['LOCAL_SEND_MAIL_PORT']}:#{defaults['LOCAL_SEND_MAIL_PORT']} + -v #{ENV['CCI_SRC']}:/c/compass-ci -v /etc/localtime:/etc/localtime:ro + -w /c/compass-ci/container/send-mail send-mail -) - -"${cmd[@]}" - -cat<<EOF -port: 11311 ---- -data='{ -"from": "email_from_addr" # default: team(a)crystal.ci -"subject": "email subject", -"to": "email_to_addr", -"body": "email message" -}' - -or - -data=" -from: email_from_addr # default: team(a)crystal.ci -subject: email subject -to: email_to_addr -body: email message" - -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" - ---- -data=" -From: email_from_addr # default: team(a)crystal.ci -To: email_to_addr -Subject: email_subject +] -mail_msg_line1 -mail_msg_line2 -... -" +cmd += ['ruby', 'send-mail.rb'] -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +system(*cmd) -- 2.23.0

2 4

[PATCH lkp-tests 1/2] jobs/block_size.yaml: add block_size.yaml [why] need test different block_size value
by Lu Kaiyi 27 Oct '20

27 Oct '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/block_size.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/block_size.yaml diff --git a/jobs/block_size.yaml b/jobs/block_size.yaml new file mode 100644 index 00000000..1e2239e5 --- /dev/null +++ b/jobs/block_size.yaml @@ -0,0 +1,21 @@ +suite: iozone +testcase: iozone +category: benchmark + +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + +disk: 2HDD + +fs: +- xfs +- ext4 +- btrfs +iosched: +- kyber + +iozone: -- 2.23.0

2 1

[PATCH lab-z9 2/2] add new host file
by Zhang Dewan 27 Oct '20

27 Oct '20

--- hosts/m | 43 ++------------------------ hosts/taishan200-2280-2s48p-256g--a102 | 1 - hosts/taishan200-2280-2s48p-256g--a103 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a104 | 23 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a105 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a106 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a107 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a16 | 20 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a35 | 21 +++++++++++++ hosts/taishan200-2280-2s48p-256g--a63 | 19 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a87 | 20 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a95 | 1 - 12 files changed, 202 insertions(+), 42 deletions(-) create mode 100644 hosts/taishan200-2280-2s48p-256g--a103 create mode 100644 hosts/taishan200-2280-2s48p-256g--a104 create mode 100644 hosts/taishan200-2280-2s48p-256g--a105 create mode 100644 hosts/taishan200-2280-2s48p-256g--a106 create mode 100644 hosts/taishan200-2280-2s48p-256g--a107 create mode 100644 hosts/taishan200-2280-2s48p-256g--a16 create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s48p-256g--a63 create mode 100644 hosts/taishan200-2280-2s48p-256g--a87 diff --git a/hosts/m b/hosts/m index b73dfb7..73ba010 100644 --- a/hosts/m +++ b/hosts/m @@ -1,40 +1,3 @@ -taishan200-2280-2s48p-256g--a11 a11 -taishan200-2280-2s48p-256g--a13 a13 -taishan200-2280-2s48p-256g--a14 a14 -taishan200-2280-2s48p-256g--a15 a15 -taishan200-2280-2s48p-256g--a17 a17 -taishan200-2280-2s48p-256g--a18 a18 -taishan200-2280-2s48p-256g--a19 a19 -taishan200-2280-2s48p-256g--a21 a21 -taishan200-2280-2s48p-256g--a22 a22 -taishan200-2280-2s48p-256g--a24 a24 -taishan200-2280-2s48p-256g--a25 a25 -taishan200-2280-2s48p-256g--a26 a26 -taishan200-2280-2s48p-256g--a27 a27 -taishan200-2280-2s48p-256g--a28 a28 -taishan200-2280-2s48p-256g--a29 a29 -taishan200-2280-2s48p-256g--a30 a30 -taishan200-2280-2s48p-256g--a31 a31 -taishan200-2280-2s48p-256g--a32 a32 -taishan200-2280-2s48p-256g--a33 a33 -taishan200-2280-2s48p-256g--a34 a34 -taishan200-2280-2s48p-256g--a52 a52 -taishan200-2280-2s48p-256g--a53 a53 -taishan200-2280-2s48p-256g--a60 a60 -taishan200-2280-2s48p-256g--a61 a61 -taishan200-2280-2s48p-256g--a62 a62 -taishan200-2280-2s48p-256g--a66 a66 -taishan200-2280-2s48p-256g--a67 a67 -taishan200-2280-2s48p-256g--a68 a68 -taishan200-2280-2s48p-256g--a69 a69 -taishan200-2280-2s48p-256g--a70 a70 -taishan200-2280-2s48p-256g--a71 a71 -taishan200-2280-2s64p-256g--a10 a10 -taishan200-2280-2s64p-256g--a3 a3 -taishan200-2280-2s64p-256g--a36 a36 -taishan200-2280-2s64p-256g--a4 a4 -taishan200-2280-2s64p-256g--a40 a40 -taishan200-2280-2s64p-256g--a41 a41 -taishan200-2280-2s64p-256g--a42 a42 -taishan200-2280-2s64p-256g--a8 a8 -taishan200-2280-2s64p-256g--a9 a9 +taishan200-2280-2s48p-256g--a16 a16 +taishan200-2280-2s48p-256g--a35 a35 +taishan200-2280-2s48p-256g--a87 a87 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-256g--a102 index e1d3f10..267a962 100644 --- a/hosts/taishan200-2280-2s48p-256g--a102 +++ b/hosts/taishan200-2280-2s48p-256g--a102 @@ -1,4 +1,3 @@ -arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 new file mode 100644 index 0000000..4afef3c --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bdc69 + - /dev/disk/by-id/scsi-350000399b88bdd85 + - /dev/disk/by-id/scsi-350000399b88bdc4d + - /dev/disk/by-id/scsi-350000399b88bdc7d +mac_addr: + - 44:67:47:c9:d8:b8 + - 44:67:47:c9:d8:b9 + - 44:67:47:c9:d8:ba + - 44:67:47:c9:d8:bb + - 44:67:47:c9:d8:bc + - 44:67:47:c9:d8:bd + - 44:67:47:c9:d8:be + - 44:67:47:c9:d8:bf + - cc:05:77:ee:b5:4f + - cc:05:77:ee:b5:50 + - cc:05:77:ee:b5:51 + - cc:05:77:ee:b5:52 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.8 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 new file mode 100644 index 0000000..3c5b807 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -0,0 +1,23 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8892431 + - /dev/disk/by-id/scsi-350000399b8892f81 + - /dev/disk/by-id/scsi-350000399b8891c49 + - /dev/disk/by-id/scsi-350000399b88930d1 +mac_addr: + - 44:67:47:d7:6d:f0 + - 44:67:47:d7:6d:f1 + - 44:67:47:d7:6d:f2 + - 44:67:47:d7:6d:f3 + - 44:67:47:d7:6d:f4 + - 44:67:47:d7:6d:f5 + - 44:67:47:d7:6d:f6 + - 44:67:47:d7:6d:f7 + - cc:05:77:ee:b9:6f + - cc:05:77:ee:b9:70 + - cc:05:77:ee:b9:71 + - cc:05:77:ee:b9:72 +model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a105 b/hosts/taishan200-2280-2s48p-256g--a105 new file mode 100644 index 0000000..3c28d2d --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a105 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b891945d + - /dev/disk/by-id/scsi-350000399b8919135 + - /dev/disk/by-id/scsi-350000399b8919699 + - /dev/disk/by-id/scsi-350000399b89190d5 +mac_addr: + - 44:67:47:d7:76:c4 + - 44:67:47:d7:76:c5 + - 44:67:47:d7:76:c6 + - 44:67:47:d7:76:c7 + - 44:67:47:d7:76:c8 + - 44:67:47:d7:76:c9 + - 44:67:47:d7:76:ca + - 44:67:47:d7:76:cb + - cc:05:77:ee:b2:8f + - cc:05:77:ee:b2:90 + - cc:05:77:ee:b2:91 + - cc:05:77:ee:b2:92 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.10 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 new file mode 100644 index 0000000..a7b44a0 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b883bf99 + - /dev/disk/by-id/scsi-350000399b883ba89 + - /dev/disk/by-id/scsi-350000399b883b94d + - /dev/disk/by-id/scsi-350000399b883b981 +mac_addr: + - 44:67:47:86:1c:30 + - 44:67:47:86:1c:31 + - 44:67:47:86:1c:32 + - 44:67:47:86:1c:33 + - 44:67:47:86:1c:34 + - 44:67:47:86:1c:35 + - 44:67:47:86:1c:36 + - 44:67:47:86:1c:37 + - cc:05:77:ee:b7:df + - cc:05:77:ee:b7:e0 + - cc:05:77:ee:b7:e1 + - cc:05:77:ee:b7:e2 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.11 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 new file mode 100644 index 0000000..9d20e57 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8917c39 + - /dev/disk/by-id/scsi-350000399b8918005 + - /dev/disk/by-id/scsi-350000399b8900a4d + - /dev/disk/by-id/scsi-350000399b8900799 +mac_addr: + - 84:46:fe:73:a2:71 + - 84:46:fe:73:a2:72 + - 84:46:fe:73:a2:73 + - 84:46:fe:73:a2:74 + - 84:46:fe:73:a2:75 + - 84:46:fe:73:a2:76 + - 84:46:fe:73:a2:77 + - 84:46:fe:73:a2:78 + - 44:a1:91:35:1a:70 + - 44:a1:91:35:1a:71 + - 44:a1:91:35:1a:72 + - 44:a1:91:35:1a:73 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.12 diff --git a/hosts/taishan200-2280-2s48p-256g--a16 b/hosts/taishan200-2280-2s48p-256g--a16 new file mode 100644 index 0000000..62cb124 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a16 @@ -0,0 +1,20 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bf271 + - /dev/disk/by-id/scsi-350000399b88bf1a9 + - /dev/disk/by-id/scsi-350000399b88bf0d5 + - /dev/disk/by-id/scsi-350000399b88bf385 +mac_addr: + - 84:46:fe:3d:a2:17 + - 84:46:fe:3d:a2:18 + - 84:46:fe:3d:a2:19 + - 84:46:fe:3d:a2:1a + - 84:46:fe:3d:a2:1b + - 84:46:fe:3d:a2:1c + - 84:46:fe:3d:a2:1d + - 84:46:fe:3d:a2:1e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..ec1e524 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,21 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-350000399886b66dd-part1 + - /dev/disk/by-id/scsi-350000399886b66dd-part2 + - /dev/disk/by-id/scsi-350000399886b66dd-part3 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 new file mode 100644 index 0000000..6d4b636 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1,19 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b0431 + - /dev/disk/by-id/scsi-350000399886adc71 + - /dev/disk/by-id/scsi-350000399886b66f5 + - /dev/disk/by-id/scsi-350000399886b6b75 +mac_addr: + - 44:67:47:c9:e5:44 + - 44:67:47:c9:e5:45 + - 44:67:47:c9:e5:46 + - 44:67:47:c9:e5:47 + - 44:67:47:c9:e5:48 + - 44:67:47:c9:e5:49 + - 44:67:47:c9:e5:4a + - 44:67:47:c9:e5:4b +model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a87 b/hosts/taishan200-2280-2s48p-256g--a87 new file mode 100644 index 0000000..b87417b --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a87 @@ -0,0 +1,20 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a6b55 + - /dev/disk/by-id/scsi-350000399886ad4a5 + - /dev/disk/by-id/scsi-350000399886a7139 + - /dev/disk/by-id/scsi-350000399886acb21 +mac_addr: + - 44:67:47:c9:eb:66 + - 44:67:47:c9:eb:67 + - 44:67:47:c9:eb:68 + - 44:67:47:c9:eb:69 + - 44:67:47:c9:eb:6a + - 44:67:47:c9:eb:6b + - 44:67:47:c9:eb:6c + - 44:67:47:c9:eb:6d +model_name: Kunpeng-920 +ipmi_ip: 9.3.8.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-256g--a95 index 213d10f..3fd8124 100644 --- a/hosts/taishan200-2280-2s48p-256g--a95 +++ b/hosts/taishan200-2280-2s48p-256g--a95 @@ -1,4 +1,3 @@ -arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -- 2.23.0

3 2

[PATCH compass-ci] fix: sched: alter 'ip' -> 'direct_ips', 'server' -> 'direct_server_ips'
by Ren Wen 27 Oct '20

27 Oct '20

[Why] 1) 'ip' -> 'direct_ips' : Cluster test script need to known direct-ip not ip. ip comes from node by command $(hostname -I | cut -d' ' -f1), it's not a direct-ip. 2) 'server' -> 'direct_server_ips' : Cluster test script uses variables 'direct_server_ips' or 'server' to connect daemon node, i perfer to 'direct_server_ips' example: iperf test: ```shell ... [ -n "$direct_server_ips" ] && server=$direct_server_ips [ -z "$server" ] && server=127.0.0.1 server=${server%% *} client_cmd="iperf3 -t $runtime -J -c $server $opt_udp" ... ``` Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index c32091f..ced0276 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -98,13 +98,13 @@ class Sched return cluster_state when "write_state" node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] + node_ip = env.params.query["direct_ips"] update_cluster_state(cluster_id, job_id, "roles", node_roles) update_cluster_state(cluster_id, job_id, "ip", node_ip) when "roles_ip" role = "server" server_ip = get_ip(cluster_id, role) - return "server=#{server_ip}" + return "direct_server_ips=#{server_ip}" end # show cluster state -- 2.23.0

1 0

[PATCH v1 compass-ci] specify the queues for the test machine to consume
by Xiao Shenwei 27 Oct '20

27 Oct '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-qemu.sh | 4 +++- providers/qemu.sh | 13 +++++++++++-- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 954d3a0..7f3aa1b 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -4,6 +4,8 @@ [[ $tbox_group ]] || tbox_group=vm-2p8g -export hostname=$tbox_group--$USER-$$ +export hostname=$tbox_group.$USER-$$ +# specify which queues will be request, use " " to separate more than 2 values +export queues="vm-2p8g.aarch64 vm-2p8g~$USER" $CCI_SRC/providers/qemu.sh diff --git a/providers/qemu.sh b/providers/qemu.sh index d97fd85..9e7627f 100755 --- a/providers/qemu.sh +++ b/providers/qemu.sh @@ -9,6 +9,7 @@ load_cci_defaults : ${hostname:="vm-1p1g-1"} +: ${queue:="vm-1p1g.$(arch)"} # unicast prefix: x2, x6, xA, xE export mac=$(echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') echo hostname: $hostname @@ -17,7 +18,15 @@ echo $mac > mac echo "arp -n | grep ${mac//-/:}" > ip.sh chmod +x ip.sh -curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" +set_mac2host() +{ + local queues + # use "," replace " " + queues=${queue/ /,} + echo $queues + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}&queues=${queues}" +} +set_mac2host del_host_mac() { @@ -33,7 +42,7 @@ trap del_host_mac EXIT tbox_group=$hostname fi - host=${tbox_group%%--*} + host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

2 6

[PATCH lkp-tests] check QEMU_EFI.fd file for multi-qemu
by Xiao Shenwei 27 Oct '20

27 Oct '20

[why] for multi-qemu job, the qemu-efi container will build error, then the QEMU_EFI.fd will not found, it caused one job will be remove from redis, but kvm can't execute it. so, this job looks like it's been executed, but not result file. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index b407251f..5f4d976b 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -7,6 +7,7 @@ multi_qemu() { + [ -f "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" ] || return export CCI_SRC=/c/compass-ci local hostname=$tbox_group--$HOSTNAME -- 2.23.0

2 1

[PATCH compass-ci 2/5] container/send-mail: mail-post.rb
by Luan Shengde 27 Oct '20

27 Oct '20

fix function invoke bugs Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-post.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 5832bfb..0b1f94a 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -22,7 +22,7 @@ post '/send_mail_yaml' do 'body' => data['body'] } - check_send_mail(mail_info) + send_mail(mail_info) end post '/send_mail_text' do -- 2.23.0

2 2

[PATCH compass-ci] doc: reset tutorial
by Li Ping 27 Oct '20

27 Oct '20

--- doc/tutorial.md | 257 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 203 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index 03f1342..aea6cc2 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -5,7 +5,7 @@ ### 概念 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github,Gitee,Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过Compass-CI，社区开发者可以将开源软件快速引入openEuler社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 ### 功能描述 @@ -30,7 +30,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - 聚合开发者测试用例 - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 + 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 - 登录环境随时调测 @@ -46,28 +46,59 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 -## 使用 Compass-CI -> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 +## 使用Compass-CI +### 前提条件 + + +- 申请账号 + + 您可以通过向**compass-ci(a)139.xn--com**,-1y7i63cba145j8lao86rn19at45crrlpsetoortax6hia2663b"**apply account**"，在正文中提供**您向开源项目提交过的commit的url地址**(我们将根据此进行审查并决定是否发放账号)，并以附件方式添加**准备用来提交任务的个人电脑的公钥**。 + 发送邮件后，将收到一封回复邮件，邮件回复内容如下： + + account_uuird: xxxxxx + SCHED_HOST: ip # 下面配置中将会使用到的ip地址 + SCHED_PORT: port # 下面配置中将会使用到的端口 + +- 构建本地可以提交job的环境 + + - 下载安装[lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) + ```bash + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + sudo make install + ``` + + - 配置lab + + - 编辑`.../lkp-tests/include/lab/z9` + ```yaml + SCHED_HOST: ip # 申请账号时回复邮件中的ip + SCHED_PORT: port # 申请账号时回复邮件中的端口 + ``` + - 新建`$HOME/.config/compass-ci/defaults/$USER.yaml` 内容如下 + ```yaml + lab: z9 + ``` - 注册自己的仓库 - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库[upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 ```bash git clone https://gitee.com/wu_fengguang/upstream-repos.git less upstream-repos/README.md ``` -### 编写 job yaml 文件 +### 编写job yaml文件 -#### job yaml 简介 -job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… +#### job yaml简介 +job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。 -#### job yaml 的结构 +#### job yaml的结构 - 标识头部 Header（必选） - 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 + 每一个job文件的开始部分，都有一些基础的描述信息，称之为yaml的Header。头部主要有suite和category，suite是测试套的名称。category是测试类型，包括benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 ```yaml suite: netperf \\测试套名称 @@ -76,7 +107,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM - 测试脚本和参数（必选） - - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 + - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 ``` $LKP_SRC/setup @@ -101,7 +132,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM SUT/schduler/部署等资源相关变量。 ```yaml - testbox: vm-2p8g + testbox: vm-hi1620-2p8g os: openeuler os_version: 20.03 os_arch: aarch64 @@ -129,52 +160,170 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM nterval: 5 ``` -**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… +**了解更多job yaml相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… ### 提交测试任务 + +使用```submit```关键字提交测试任务，以netperf为例，命令如下。 +```bash +submit netperf.yaml +``` + + +submit命令更多参数设置方法如下。 +``` +- Usage: submit [options] jobs... + submit test jobs to scheduler +- options: + - s, --set 'KEY: VALUE' #add YAML hash to job + - o, --output DIR #save job yaml to DIR + - a, --auto-define-files #auto add define_files + - c, --connect #auto connect to the host + - m, --monitor # monitor job status: use -m 'KEY: VALUE' to add rule" +``` + + ### 查看测试结果 + +- 生成与存储测试结果 + 测试执行机完成测试任务后，将结果保存到日志文件中，并上传至服务器，按照```$suite/$tbox_group/$date/job_id```的目录结构存储在本地```/srv/result```目录。 + extract-stats 服务将提取本地日志文件中的数据，生成与日志文件对应的json文件，并将汇总后的结果存储到obs数据库(ES)对应id的job中。 + + +- web页面查看结果: + - 点击链接查看obs数据库（ES）中的结果：https://compass-ci.openeuler.org/jobs + + - 点击链接查看文件中的结果,示例：http://124.90.34.227:11300/result/iperf/dc-2g--xiao/2020-09-21/crystal.83385/，文件分为两类： + + - 由测试执行机上传的日志文件： + boot-time, diskstats.gz, interrupts.gz, ...等。 + + - 由extract-stats服务生成的json文件： + boot-time.json, diskstats.json, interrupts.json,stats.json, ... 等。 + json 文件对应每一个日志文件提取后的结果，stats.json为汇总后的结果。 + + ### 比较测试结果 -### 提交 borrow 任务 +测试完成后，Compass-CI通过条件查询数据并将数据合并为多个矩阵，计算每个矩阵的平均值和标准差，以比较测试用例在特定维度的性能变化。最后，将比较结果打印输出。 + + +可以通过如下两种方式比较测试结果： + +- 通过compare web比较 + Compass CI/compare + +- 通过命令行比较 + ``` + compare conditions -d dimension + ``` + +示例比较测试结果如下。 +``` + os=openeuler/os_arch=aarch64/tbox_group=vm-hi1620-2p8g + + + 20 1 metric + -------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 8:21 -38.1% 0:1 last_state.daemon.sshd.exit_code.1 + 1:21 -4.8% 0:1 last_state.daemon.sshd.exit_code.2 + 1:21 -4.8% 0:1 last_state.setup.disk.exit_code.1 + 1:21 -4.8% 0:1 last_state.test.dbench.exit_code.99 + 1:21 -4.8% 0:1 last_state.test.email.exit_code.7 + + + 20 1 metric + -------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ + 0.02 ± 265% +2259.8% 0.57 mpstat.cpu.all.soft% + 0.48 ± 299% +1334.9% 6.90 mpstat.cpu.all.sys% + 2760.71 ± 164% +292.6% 10838.00 proc-vmstat.nr_dirty_background_threshold + 5522.10 ± 164% +292.6% 21680.00 proc-vmstat.nr_dirty_threshold + +``` + + +### 提交borrow任务 + +borrow任务是指通过提交任务的方式申请环境，提交borrow任务的yaml文件请参考lkp-tests/jobs/borrow-1h.yaml。 + +1. yaml文件配置说明： + + - 必填字段： + ``` + sshd: + pub_key \\将用户的公钥信息添加到job中 + email \\ 配置用户邮箱地址 + runtime \\申请环境的使用时间 h/d/w + testbox \\申请环境的规格 + ``` + - 选填字段： + ``` + os \\申请环境的操作系统参数 + os_arch + os_version + ``` + +2. 提交任务 + + 执行命令```submit -m -c borrow-1h.yaml```，可以提交borrow任务并自动ssh连接到申请的环境当中。 + ### 提交 bisect 任务 +bisect 任务可以找到首次在git repo中引入问题的commit信息。提交bisect任务的yaml文件请参考 $LKP_SRC/jobs/bisect.yaml。 + +1. yaml文件配置说明： + - 必填字段： + ``` + bisect: + job_id: \\提交job的job_id + error_id: \\在Compass-CI网页上搜索$job_id，得到$error_ids，从$error_ids中选择一个$error_id。 + ``` +2. 提交任务 + 执行命令```submit bisect.yaml```，提交成功后会收到通知邮件。 + + + ## 高级功能 -### 添加 OS 支持 +### 添加OS支持 -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 +制作一个initramfs启动的cgz镜像，当系统内核启动时，直接从打包的cgz镜像中导出rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的I/O性能。 #### 操作步骤 - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs + 1. 获取对应os版本的rootfs（以openEuler为例） + - 通过docker获取rootfs - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 + 1) 下载openEuler官方提供的docker镜像压缩包 ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… ``` - 2) 加载 docker 镜像 + 2) 加载docker镜像 ```bash docker load -i openEuler-docker.aarch64 ``` - 3) 启动 openEuler 容器 + 3) 启动openEuler容器 ```bash docker run -id openeuler-20.03-lts ``` - 4) 拷贝 docker 的 rootfs + 4) 拷贝docker的rootfs ```bash docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) + - 通过qemu.img(qcow2格式)获取rootfs (以openEuler为例) - 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 + 1) 下载openEuler官方网站提供的qcow2格式镜像 ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` @@ -184,22 +333,22 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` 2. 定制rootfs - 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) + 1. 使用chroot命令切换到rootfs中(此步骤需要root权限) ```bash chroot openEuler-rootfs ``` 2. 根据个人需要安装并配置服务 - a. 修改 root 密码 - b. 配置 ssh 服务 + a. 修改root密码 + b. 配置ssh服务 c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： + d. 如果使用docker制作osimage还需要以下操作： > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核rpm包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 + > 2. 从centos官方网站下载内核rpm包 + > 3. 使用yum进行安装 + > 4. 删除docker环境变量文件 - 3. 退出 rootfs，并打包 + 3. 退出rootfs，并打包 ```bash cd $rootfs find . | coip -o -Hnewc |gzip -9 > $os_name.cgz @@ -218,7 +367,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM - 解决方法 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 + 2）内核文件权限不足，给予644权限。 2. 系统运行缓慢 @@ -227,7 +376,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 打包镜像体积过大，会消耗很大内存 - 解决方法 - 建议用户根据具体需要对 rootfs 进行裁剪。 + 建议用户根据具体需要对rootfs进行裁剪。 ### 添加测试用例 @@ -238,13 +387,13 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 1. 概述： - 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 - PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 + 使用PKGBUILD完成上游项目源代码的编译构建，同一项目可以使用不同PKGBUILD完成项目编译构建过程。 + PKGBUILD相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD 2. 步骤： 1) 注册仓库 - 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + 如果您想在git push的时候, 自动触发测试, 那么需要把您的公开git url添加到如下仓库[upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) ```bash git clone https://gitee.com/wu_fengguang/upstream-repos.git less upstream-repos/README.md @@ -254,17 +403,17 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 项目代码更新后自动触发构建任务，无需其他操作。 3) 查看结果 - web: https://compass-ci.openeuler.org/jobs。 + web: https://compass-ci.openeuler.org/jobs 4) 备注 - 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + 目前提供archlinux下已有PKGBUILD项目构建测试，暂不支持自主导入PKGBUILD文件。 + PKGBUILD存放路径： /srv/git/archlinux/*/*/PKGBUILD ### depends - 依赖包构建 - 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 + 流程: 以下默认使用debian:sid版本进行,如果使用其他OS跑测试用例，则需要在lkp-tests/distro/adaptation/下将包名以键值对的形式写入对应OS的文件中 > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 @@ -280,40 +429,40 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM -### 本地搭建 compass-ci 服务器节点 +### 本地搭建compass-ci 服务器节点 -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 +在openEuler系统一键部署compass-ci环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 #### 准备工作 - 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 8GB - CPU：64 nuclear (建议) + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 8GB + CPU：64 nuclear (建议) 硬盘：>= 500G - 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G 网络：可以访问互联网 >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + >openEuler系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/I… #### 操作指导 -1. 登录 openEuler 系统 +1. 登录openEuler系统 2. 创建工作目录并设置文件权限 ```bash mkdir demo && cd demo && umask 002 ``` -3. 克隆 compass-ci 项目代码到 demo 目录 +3. 克隆compass-ci项目代码到demo目录 ```bash git clone https://gitee.com/wu_fengguang/compass-ci.git ``` -4. 执行一键部署脚本 install-tiny +4. 执行一键部署脚本install-tiny ```bash cd compass-ci/sparrow && ./install-tiny ``` -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. [How] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..c5f29d74 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,4 +1,5 @@ #!/bin/sh +# - block_size ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. @@ -7,7 +8,11 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + if [ -n "$block_size" ]; then + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" + else + cmd="iozone -a -f $mnt/iozonetest" + fi log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. [How] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..c5f29d74 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,4 +1,5 @@ #!/bin/sh +# - block_size ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. @@ -7,7 +8,11 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + if [ -n "$block_size" ]; then + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" + else + cmd="iozone -a -f $mnt/iozonetest" + fi log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

1 0

[PATCH v1 compass-ci] based on queue field add the job into specify queue
by Xiao Shenwei 27 Oct '20

27 Oct '20

use queue replace tbox_group Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 692180f..d4929cd 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -30,12 +30,15 @@ class Sched mac.gsub(":", "-") end - def set_host_mac(mac : String, hostname : String) + def set_host_mac(mac : String, hostname : String, queues : String) + puts "queues: #{queues}" @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + @redis.hash_set("sched/mac2queues", normalize_mac(mac), queues) end def del_host_mac(mac : String) @redis.hash_del("sched/mac2host", normalize_mac(mac)) + @redis.hash_del("sched/mac2queues", normalize_mac(mac)) end # return: @@ -251,24 +254,24 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - tbox_group = job.tbox_group + queue = job.queue return [{ "job_id" => "0", - "message" => "get tbox group failed", + "message" => "get queue failed", "job_state" => "submit", - }] unless tbox_group + }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? - tbox_group += "#{job.get_uuid_tag}" + queue += "#{job.get_uuid_tag}" else - tbox_group = "#{tbox_group}/idle" + queue = "#{queue}/idle" end - job_id = add_task(tbox_group, job.lab) + job_id = add_task(queue, job.lab) return [{ "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", }] unless job_id @@ -372,7 +375,7 @@ class Sched hostname = @redis.hash_get("sched/mac2host", normalize_mac(api_param)) if hostname.nil? # auto name new/unknown machine hostname = "sut-#{api_param}" - set_host_mac(api_param, hostname) + set_host_mac(api_param, hostname, "") # auto submit a job to collect the host information # grub hostname is link with ":", like "00:01:02:03:04:05" -- 2.23.0

2 2

[PATCH compass-ci] sched: mount default config dir for idle job
by Cao Xueliang 27 Oct '20

27 Oct '20

[why] There no default config yaml in scheduler container, when submit idle job use the submit script. [how] Mount default config dir in scheduler container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 3 +++ src/scheduler/jobfile_operate.cr | 10 +++------- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index d61616a..6a2a4b0 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -45,6 +45,8 @@ SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' + docker_rm "s001-alpine-#{SCHED_PORT}" bind_list = [] @@ -70,6 +72,7 @@ cmd = %W[ -e FLUENTD_SERVER_PORT=#{FLUENTD_SERVER_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index e0e298f..d3c26c5 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -202,14 +202,10 @@ module Jobfile::Operate end def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) - job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", - "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "GIT_SERVER=#{ENV["GIT_SERVER"]}", - "lab=#{ENV["lab"]}"] - - job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " - cmd += job_fields.join(" ") + if other_parameters + cmd += other_parameters.join(" ") + end cmd += " -s '#{override_parameter}' #{job_file}" puts `#{cmd}` end -- 2.23.0

1 0

[PATCH v3 compass-ci] add queue field into job.yaml
by Xiao Shenwei 27 Oct '20

27 Oct '20

[why] after one job submit, it will be add specified queue before: use tbox_group to discriminate now : use queue field to discriminate so, we should add this field into job.yaml it's come from user's specify(job.yaml) or dafault Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/job.cr | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8f5dd36..0376f9f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -73,6 +73,7 @@ class Job tbox_group testbox lab + queue initrd_pkg initrd_deps initrds_uri @@ -139,6 +140,7 @@ class Job set_kernel_params() set_lkp_server() set_sshr_port() + set_queue() end private def append_init_field @@ -199,6 +201,16 @@ class Job self["result_service"] = "raw_upload" end + private def set_queue + return if @hash.has_key?("queue") + + # set default value + self["queue"] = tbox_group + if tbox_group.to_s.starts_with?(/(vm|dc).*/) + self["queue"] = "#{tbox_group}.#{arch}" + end + end + # if not assign tbox_group, set it to a match result from testbox # ?if job special testbox, should we just set tbox_group=textbox private def update_tbox_group_from_testbox -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue api: use same client for 1 server
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] interaction to taskqueue will be frequently, so keep same client as possible. - btw: caller, that do not need keep-alive with taskqueue, should set header["Connection"]="close", or active close the connection. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/taskqueue_api.cr | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/src/lib/taskqueue_api.cr b/src/lib/taskqueue_api.cr index 5a83a15..a8bbf16 100644 --- a/src/lib/taskqueue_api.cr +++ b/src/lib/taskqueue_api.cr @@ -4,25 +4,33 @@ require "json" require "http/client" +def loop_till_connectable + loop do + begin + return yield + rescue ex + puts "#{ex}, retrying in 5s..." + sleep(5) + end + end +end + class TaskQueueAPI def initialize - @port = ENV.has_key?("TASKQUEUE_PORT") ? ENV["TASKQUEUE_PORT"].to_i32 : 3060 - @host = ENV.has_key?("TASKQUEUE_HOST") ? ENV["TASKQUEUE_HOST"] : "172.17.0.1" + port = ENV.has_key?("TASKQUEUE_PORT") ? ENV["TASKQUEUE_PORT"].to_i32 : 3060 + host = ENV.has_key?("TASKQUEUE_HOST") ? ENV["TASKQUEUE_HOST"] : "172.17.0.1" + @client = HTTP::Client.new(host, port: port) end def add_task(service_queue_path : String, task : JSON::Any) params = HTTP::Params.encode({"queue" => service_queue_path}) - client = HTTP::Client.new(@host, port: @port) - response = client.post("/add?" + params, body: task.to_json) - client.close + response = loop_till_connectable { @client.post("/add?" + params, body: task.to_json) } arrange_response(response) end def query_keys(service_key_with_wild_char : String) params = HTTP::Params.encode({"queue" => service_key_with_wild_char}) - client = HTTP::Client.new(@host, port: @port) - response = client.get("/keys?" + params) - client.close + response = loop_till_connectable { @client.get("/keys?" + params) } arrange_response(response) end @@ -42,9 +50,7 @@ class TaskQueueAPI end private def response_put_api(cmd : String, params : String) - client = HTTP::Client.new(@host, port: @port) - response = client.put("/#{cmd}?" + params) - client.close + response = loop_till_connectable { @client.put("/#{cmd}?" + params) } arrange_response(response) end -- 2.23.0

1 0

[PATCH compass-ci] fix: scheduler: close connection after API call
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] most client call to scheduler do not need keep-alive. when API call is finished, close the connection. [how] set response header["Connection"]="close" Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/scheduler/scheduler.cr | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5befcf9..371e4ed 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -39,6 +39,7 @@ module Scheduler # echo alive get "/" do |env| + env.response.headers["Connection"] = "close" debug_message(env, "Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{VERSION}" end @@ -49,6 +50,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| + env.response.headers["Connection"] = "close" response = sched.find_job_boot(env) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @@ -59,6 +61,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| + env.response.headers["Connection"] = "close" response = sched.find_next_job_boot(env) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @@ -71,6 +74,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| + env.response.headers["Connection"] = "close" job_messages = sched.submit_job(env) job_messages.each do |job_message| @@ -82,6 +86,7 @@ module Scheduler # file download server get "/job_initrd_tmpfs/:job_id/:job_package" do |env| + env.response.headers["Connection"] = "close" job_id = env.params.url["job_id"] job_package = env.params.url["job_package"] file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] @@ -98,6 +103,7 @@ module Scheduler # # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| + env.response.headers["Connection"] = "close" if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) sched.set_host_mac(client_mac, client_hostname) @@ -109,6 +115,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| + env.response.headers["Connection"] = "close" if client_mac = env.params.query["mac"]? sched.del_host_mac(client_mac) @@ -124,6 +131,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| + env.response.headers["Connection"] = "close" puts sched.update_job_parameter(env) "Done" @@ -147,6 +155,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| + env.response.headers["Connection"] = "close" response = sched.request_cluster_state(env) debug_message(env, response) @@ -158,6 +167,7 @@ module Scheduler # /~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job.yaml&job_id=40 # curl "http://localhost:3000/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job…" get "/~lkp/cgi-bin/lkp-post-run" do |env| + env.response.headers["Connection"] = "close" # get job_id from request job_id = env.params.query["job_id"]? if job_id @@ -168,12 +178,14 @@ module Scheduler end get "/~lkp/cgi-bin/lkp-wtmp" do |env| + env.response.headers["Connection"] = "close" puts sched.update_tbox_wtmp(env) "Done" end get "/~lkp/cgi-bin/report_ssh_port" do |env| + env.response.headers["Connection"] = "close" testbox = env.params.query["tbox_name"] ssh_port = env.params.query["ssh_port"].to_s job_id = env.params.query["job_id"].to_s -- 2.23.0

1 0

[PATCH v2 compass-ci] /lib/es_query.rb: add range query
by Zhang Yuhang 27 Oct '20

27 Oct '20

[why] Some queries need to limit the range of fields. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..037213d 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -36,12 +36,27 @@ class ESQuery end end +# Range Query Example: +# range = { +# start_time: { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# end_time: { +# gt: '2020-09-10 01:52:00', +# lt: '2020-09-10 01:54:00' +# } +# } +# items['range'] = range +# build_mutli_field_subquery_body(items) def build_mutli_field_subquery_body(items) query_fields = [] items.each do |key, value| if value.is_a?(Array) inner_query = build_multi_field_or_query_body(key, value) query_fields.push({ bool: { should: inner_query } }) + elsif key.to_s == 'range' + query_fields.concat(value.map { |k, v| { range: { k => v } } }) else query_fields.push({ term: { key => value } }) end -- 2.23.0

2 2

[PATCH lab-z9 3/3] rename 3 files for memory change to 512G
by Zhang Dewan 27 Oct '20

27 Oct '20

--- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 ...200-2280-2s48p-256g--a105 => taishan200-2280-2s48p-512g--a105} | 0 ...an200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) rename hosts/{taishan200-2280-2s48p-256g--a105 => taishan200-2280-2s48p-512g--a105} (100%) rename hosts/{taishan200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 diff --git a/hosts/taishan200-2280-2s48p-256g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a105 rename to hosts/taishan200-2280-2s48p-512g--a105 diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a95 rename to hosts/taishan200-2280-2s48p-512g--a95 -- 2.23.0

1 0

[PATCH lab-z9 2/3] add new host file
by Zhang Dewan 27 Oct '20

27 Oct '20

--- hosts/m | 43 ++------------------------ hosts/taishan200-2280-2s48p-256g--a102 | 1 - hosts/taishan200-2280-2s48p-256g--a103 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a104 | 23 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a105 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a106 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a107 | 24 ++++++++++++++ hosts/taishan200-2280-2s48p-256g--a16 | 20 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a35 | 21 +++++++++++++ hosts/taishan200-2280-2s48p-256g--a63 | 19 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a87 | 20 ++++++++++++ hosts/taishan200-2280-2s48p-256g--a95 | 1 - 12 files changed, 202 insertions(+), 42 deletions(-) create mode 100644 hosts/taishan200-2280-2s48p-256g--a103 create mode 100644 hosts/taishan200-2280-2s48p-256g--a104 create mode 100644 hosts/taishan200-2280-2s48p-256g--a105 create mode 100644 hosts/taishan200-2280-2s48p-256g--a106 create mode 100644 hosts/taishan200-2280-2s48p-256g--a107 create mode 100644 hosts/taishan200-2280-2s48p-256g--a16 create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s48p-256g--a63 create mode 100644 hosts/taishan200-2280-2s48p-256g--a87 diff --git a/hosts/m b/hosts/m index b73dfb7..73ba010 100644 --- a/hosts/m +++ b/hosts/m @@ -1,40 +1,3 @@ -taishan200-2280-2s48p-256g--a11 a11 -taishan200-2280-2s48p-256g--a13 a13 -taishan200-2280-2s48p-256g--a14 a14 -taishan200-2280-2s48p-256g--a15 a15 -taishan200-2280-2s48p-256g--a17 a17 -taishan200-2280-2s48p-256g--a18 a18 -taishan200-2280-2s48p-256g--a19 a19 -taishan200-2280-2s48p-256g--a21 a21 -taishan200-2280-2s48p-256g--a22 a22 -taishan200-2280-2s48p-256g--a24 a24 -taishan200-2280-2s48p-256g--a25 a25 -taishan200-2280-2s48p-256g--a26 a26 -taishan200-2280-2s48p-256g--a27 a27 -taishan200-2280-2s48p-256g--a28 a28 -taishan200-2280-2s48p-256g--a29 a29 -taishan200-2280-2s48p-256g--a30 a30 -taishan200-2280-2s48p-256g--a31 a31 -taishan200-2280-2s48p-256g--a32 a32 -taishan200-2280-2s48p-256g--a33 a33 -taishan200-2280-2s48p-256g--a34 a34 -taishan200-2280-2s48p-256g--a52 a52 -taishan200-2280-2s48p-256g--a53 a53 -taishan200-2280-2s48p-256g--a60 a60 -taishan200-2280-2s48p-256g--a61 a61 -taishan200-2280-2s48p-256g--a62 a62 -taishan200-2280-2s48p-256g--a66 a66 -taishan200-2280-2s48p-256g--a67 a67 -taishan200-2280-2s48p-256g--a68 a68 -taishan200-2280-2s48p-256g--a69 a69 -taishan200-2280-2s48p-256g--a70 a70 -taishan200-2280-2s48p-256g--a71 a71 -taishan200-2280-2s64p-256g--a10 a10 -taishan200-2280-2s64p-256g--a3 a3 -taishan200-2280-2s64p-256g--a36 a36 -taishan200-2280-2s64p-256g--a4 a4 -taishan200-2280-2s64p-256g--a40 a40 -taishan200-2280-2s64p-256g--a41 a41 -taishan200-2280-2s64p-256g--a42 a42 -taishan200-2280-2s64p-256g--a8 a8 -taishan200-2280-2s64p-256g--a9 a9 +taishan200-2280-2s48p-256g--a16 a16 +taishan200-2280-2s48p-256g--a35 a35 +taishan200-2280-2s48p-256g--a87 a87 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-256g--a102 index e1d3f10..267a962 100644 --- a/hosts/taishan200-2280-2s48p-256g--a102 +++ b/hosts/taishan200-2280-2s48p-256g--a102 @@ -1,4 +1,3 @@ -arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 new file mode 100644 index 0000000..4afef3c --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bdc69 + - /dev/disk/by-id/scsi-350000399b88bdd85 + - /dev/disk/by-id/scsi-350000399b88bdc4d + - /dev/disk/by-id/scsi-350000399b88bdc7d +mac_addr: + - 44:67:47:c9:d8:b8 + - 44:67:47:c9:d8:b9 + - 44:67:47:c9:d8:ba + - 44:67:47:c9:d8:bb + - 44:67:47:c9:d8:bc + - 44:67:47:c9:d8:bd + - 44:67:47:c9:d8:be + - 44:67:47:c9:d8:bf + - cc:05:77:ee:b5:4f + - cc:05:77:ee:b5:50 + - cc:05:77:ee:b5:51 + - cc:05:77:ee:b5:52 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.8 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 new file mode 100644 index 0000000..3c5b807 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -0,0 +1,23 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8892431 + - /dev/disk/by-id/scsi-350000399b8892f81 + - /dev/disk/by-id/scsi-350000399b8891c49 + - /dev/disk/by-id/scsi-350000399b88930d1 +mac_addr: + - 44:67:47:d7:6d:f0 + - 44:67:47:d7:6d:f1 + - 44:67:47:d7:6d:f2 + - 44:67:47:d7:6d:f3 + - 44:67:47:d7:6d:f4 + - 44:67:47:d7:6d:f5 + - 44:67:47:d7:6d:f6 + - 44:67:47:d7:6d:f7 + - cc:05:77:ee:b9:6f + - cc:05:77:ee:b9:70 + - cc:05:77:ee:b9:71 + - cc:05:77:ee:b9:72 +model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a105 b/hosts/taishan200-2280-2s48p-256g--a105 new file mode 100644 index 0000000..3c28d2d --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a105 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b891945d + - /dev/disk/by-id/scsi-350000399b8919135 + - /dev/disk/by-id/scsi-350000399b8919699 + - /dev/disk/by-id/scsi-350000399b89190d5 +mac_addr: + - 44:67:47:d7:76:c4 + - 44:67:47:d7:76:c5 + - 44:67:47:d7:76:c6 + - 44:67:47:d7:76:c7 + - 44:67:47:d7:76:c8 + - 44:67:47:d7:76:c9 + - 44:67:47:d7:76:ca + - 44:67:47:d7:76:cb + - cc:05:77:ee:b2:8f + - cc:05:77:ee:b2:90 + - cc:05:77:ee:b2:91 + - cc:05:77:ee:b2:92 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.10 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 new file mode 100644 index 0000000..a7b44a0 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b883bf99 + - /dev/disk/by-id/scsi-350000399b883ba89 + - /dev/disk/by-id/scsi-350000399b883b94d + - /dev/disk/by-id/scsi-350000399b883b981 +mac_addr: + - 44:67:47:86:1c:30 + - 44:67:47:86:1c:31 + - 44:67:47:86:1c:32 + - 44:67:47:86:1c:33 + - 44:67:47:86:1c:34 + - 44:67:47:86:1c:35 + - 44:67:47:86:1c:36 + - 44:67:47:86:1c:37 + - cc:05:77:ee:b7:df + - cc:05:77:ee:b7:e0 + - cc:05:77:ee:b7:e1 + - cc:05:77:ee:b7:e2 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.11 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 new file mode 100644 index 0000000..9d20e57 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8917c39 + - /dev/disk/by-id/scsi-350000399b8918005 + - /dev/disk/by-id/scsi-350000399b8900a4d + - /dev/disk/by-id/scsi-350000399b8900799 +mac_addr: + - 84:46:fe:73:a2:71 + - 84:46:fe:73:a2:72 + - 84:46:fe:73:a2:73 + - 84:46:fe:73:a2:74 + - 84:46:fe:73:a2:75 + - 84:46:fe:73:a2:76 + - 84:46:fe:73:a2:77 + - 84:46:fe:73:a2:78 + - 44:a1:91:35:1a:70 + - 44:a1:91:35:1a:71 + - 44:a1:91:35:1a:72 + - 44:a1:91:35:1a:73 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.12 diff --git a/hosts/taishan200-2280-2s48p-256g--a16 b/hosts/taishan200-2280-2s48p-256g--a16 new file mode 100644 index 0000000..62cb124 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a16 @@ -0,0 +1,20 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bf271 + - /dev/disk/by-id/scsi-350000399b88bf1a9 + - /dev/disk/by-id/scsi-350000399b88bf0d5 + - /dev/disk/by-id/scsi-350000399b88bf385 +mac_addr: + - 84:46:fe:3d:a2:17 + - 84:46:fe:3d:a2:18 + - 84:46:fe:3d:a2:19 + - 84:46:fe:3d:a2:1a + - 84:46:fe:3d:a2:1b + - 84:46:fe:3d:a2:1c + - 84:46:fe:3d:a2:1d + - 84:46:fe:3d:a2:1e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..ec1e524 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,21 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-350000399886b66dd-part1 + - /dev/disk/by-id/scsi-350000399886b66dd-part2 + - /dev/disk/by-id/scsi-350000399886b66dd-part3 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 new file mode 100644 index 0000000..6d4b636 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1,19 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b0431 + - /dev/disk/by-id/scsi-350000399886adc71 + - /dev/disk/by-id/scsi-350000399886b66f5 + - /dev/disk/by-id/scsi-350000399886b6b75 +mac_addr: + - 44:67:47:c9:e5:44 + - 44:67:47:c9:e5:45 + - 44:67:47:c9:e5:46 + - 44:67:47:c9:e5:47 + - 44:67:47:c9:e5:48 + - 44:67:47:c9:e5:49 + - 44:67:47:c9:e5:4a + - 44:67:47:c9:e5:4b +model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a87 b/hosts/taishan200-2280-2s48p-256g--a87 new file mode 100644 index 0000000..b87417b --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a87 @@ -0,0 +1,20 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a6b55 + - /dev/disk/by-id/scsi-350000399886ad4a5 + - /dev/disk/by-id/scsi-350000399886a7139 + - /dev/disk/by-id/scsi-350000399886acb21 +mac_addr: + - 44:67:47:c9:eb:66 + - 44:67:47:c9:eb:67 + - 44:67:47:c9:eb:68 + - 44:67:47:c9:eb:69 + - 44:67:47:c9:eb:6a + - 44:67:47:c9:eb:6b + - 44:67:47:c9:eb:6c + - 44:67:47:c9:eb:6d +model_name: Kunpeng-920 +ipmi_ip: 9.3.8.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-256g--a95 index 213d10f..3fd8124 100644 --- a/hosts/taishan200-2280-2s48p-256g--a95 +++ b/hosts/taishan200-2280-2s48p-256g--a95 @@ -1,4 +1,3 @@ -arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: comsume exists queue
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] when there has no job for a testbox, scheduler will consume 4-5 times testbox*1 tbox_group*1 tbox*1 tbox_group/idle*(1|2) [how] query taskqueue the queue-name first. if the queue-name not exist, then there should be no job for this queue. submit idle job at background, let the consume not wait for submit idle job finished. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/sched.cr | 52 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7215df5..8fbe9aa 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -409,19 +409,26 @@ class Sched tbox_group = JobHelper.match_tbox_group(testbox) tbox = tbox_group.partition("--")[0] - boxes = [testbox, tbox_group, tbox] + queue_list = query_consumable_keys(tbox) + + boxes = ["sched/" + testbox, + "sched/" + tbox_group, + "sched/" + tbox, + "sched/" + tbox_group + "/idle"] boxes.each do |box| + next if queue_list.select(box).size == 0 count.times do - job = prepare_job("sched/#{box}", testbox) + job = prepare_job(box, testbox) return job if job sleep(1) unless count == 1 end end - # when find no job at "sched/#{tbox_group}" - # try to get from "sched/#{tbox_group}/idle" - return get_idle_job(tbox_group, testbox) + # when find no job, auto submit idle job at background + spawn { auto_submit_idle_job(tbox_group) } + + return nil end private def prepare_job(queue_name, testbox) @@ -571,4 +578,39 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end + + private def query_consumable_keys(shortest_queue_name) + keys = [] of String + search = "sched/" + shortest_queue_name + "*" + response = @task_queue.query_keys(search) + + return keys unless response[0] == 200 + + key_list = JSON.parse(response[1].to_json).as_a + + # add consumable keys + key_list.each do |key| + queue_name = consumable_key?("#{key}") + keys << queue_name if queue_name + end + + return keys + end + + private def consumable_key?(key_name) + if key_name =~ /(.*)\/(.*)$/ + case $2 + when "in_process" + return nil + when "ready" + return $1 + when "idle" + return key_name + else + return key_name + end + end + + return nil + end end -- 2.23.0

4 6

[PATCH compass-ci] fix: container: avoid bind to all ip
by Tong Qunfeng 27 Oct '20

27 Oct '20

[why] default docker "-p port:port" will bind to all ips. - "-p 0.0.0.0:port:port" will also bind to all ips. - if exists ipv6 ip, it will bind to ipv6 address too. - if bind to ipv6 address, there will be tcp6 socket in TIME_WAIT no need bind to all ips. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- container/scheduler/start | 9 +++++++-- container/taskqueue/start | 9 ++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index 305916e..d61616a 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -47,14 +47,19 @@ FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_PORT}" +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + cmd = %W[ docker run --restart=always --name s001-alpine-#{SCHED_PORT} -d -u 1090:1090 - -p #{SCHED_PORT}:#{SCHED_PORT} -] + env + %W[ +] + bind_list + env + %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} diff --git a/container/taskqueue/start b/container/taskqueue/start index 4760b7d..bdd5e7c 100755 --- a/container/taskqueue/start +++ b/container/taskqueue/start @@ -16,19 +16,26 @@ defaults = relevant_defaults(names) env = docker_env(defaults) service_port = defaults['TASKQUEUE_PORT'] || '3060' +service_host = defaults['TASKQUEUE_HOST'] || '172.17.0.1' service_name = 'taskqueue' service_image = "#{service_name}-alpine:v0.001-#{service_port}" docker_rm "#{service_name}-#{service_port}" +bind_list = [] +[service_host, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{service_port}:#{service_port}" +end + cmd = %W[ docker run --name #{service_name}-#{service_port} --restart=always -d -u 1090:1090 - -p #{service_port}:#{service_port} ] +cmd += bind_list cmd += env cmd += %W[ -v /etc/localtime:/etc/localtime:ro -- 2.23.0

1 0

对于compass-ci的两个问题请教
by 邵薛华 27 Oct '20

27 Oct '20

您好，这里有两个问题麻烦请教一下您1.compass-ci之后的开发计划如何，对于compass-ci以后的功能延伸有哪些。 2.compass-ci在现在的开发程度来说，是否只能针对open Euler-20.03进行测试，能否对open Euler-20.09，或者我想通过compass-ci是否可以对基于openEuler延伸开发出来的其他.iso进行一次测试。

1 0

[PATCH compass-ci 5/5] container/send-mail: delete old useless files
by Luan Shengde 27 Oct '20

27 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/conf/sources.list | 9 ----- container/send-mail/mail-post.rb | 48 --------------------------- 2 files changed, 57 deletions(-) delete mode 100644 container/send-mail/conf/sources.list delete mode 100755 container/send-mail/mail-post.rb diff --git a/container/send-mail/conf/sources.list b/container/send-mail/conf/sources.list deleted file mode 100644 index 8179e01..0000000 --- a/container/send-mail/conf/sources.list +++ /dev/null @@ -1,9 +0,0 @@ -deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib - -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb deleted file mode 100755 index 4eecf6e..0000000 --- a/container/send-mail/mail-post.rb +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -require 'sinatra' -require 'json' -require 'yaml' -require 'open3' -require_relative 'send-mail.rb' - -set :bind, '0.0.0.0' -set :port, 11311 - -post '/send_mail_yaml' do - data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash - - mail_info = { - 'references' => data['references'] || '', - 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] - } - check_send_mail(mail_info) -end - -post '/send_mail_text' do - data = Mail.read_from_string(request.body.read) - - mail_info = { - 'references' => data.references || '', - 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded - } - check_send_mail(mail_info) -end - -def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? - - send_mail(mail_info) -end -- 2.23.0

1 0

[PATCH compass-ci 1/5] container/send-mail: send-mail.rb
by Luan Shengde 27 Oct '20

27 Oct '20

setup smtp config for email server invoke mail-post to hand the send mail request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..436736c 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -4,25 +4,14 @@ # frozen_string_literal: true require 'mail' +require 'sinatra' +require "#{ENV['CCI_SRC']}/lib/mail-post" -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp -# setup smtp config smtp = { address: mail_server, enable_starttls_auto: false } Mail.defaults { delivery_method :smtp, smtp } - -# send mail -def send_mail(mail_info) - mail = Mail.new do - references mail_info['references'] - from mail_info['from'] - subject mail_info['subject'] - to mail_info['to'] - body mail_info['body'] - end - mail.deliver! -end -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add cci-makepkg.yaml
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] Now we run cci-makepkg to make packages. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- jobs/cci-makepkg.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/cci-makepkg.yaml diff --git a/jobs/cci-makepkg.yaml b/jobs/cci-makepkg.yaml new file mode 100644 index 00000000..5d437969 --- /dev/null +++ b/jobs/cci-makepkg.yaml @@ -0,0 +1,5 @@ +suite: cci-makepkg +category: functional + +cci-makepkg: + benchmark: -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add cci-depends.yaml
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] Now we run cci-depends to make pkg depends. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- jobs/cci-depends.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/cci-depends.yaml diff --git a/jobs/cci-depends.yaml b/jobs/cci-depends.yaml new file mode 100644 index 00000000..0867068d --- /dev/null +++ b/jobs/cci-depends.yaml @@ -0,0 +1,5 @@ +suite: cci-depends +category: functional + +cci-depends: + benchmark: -- 2.23.0

1 0

[PATCH compass-ci] /lib/es_query.rb: add range query
by Zhang Yuhang 27 Oct '20

27 Oct '20

[why] Some queries need to limit the range of fields. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..3cf9852 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -42,6 +42,8 @@ def build_mutli_field_subquery_body(items) if value.is_a?(Array) inner_query = build_multi_field_or_query_body(key, value) query_fields.push({ bool: { should: inner_query } }) + elsif key.to_s == 'range' + query_fields.push({ range: value }) else query_fields.push({ term: { key => value } }) end -- 2.23.0

3 6

[PATCH v4 lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. [How] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..c5f29d74 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,4 +1,5 @@ #!/bin/sh +# - block_size ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. @@ -7,7 +8,11 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + if [ -n "$block_size" ]; then + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" + else + cmd="iozone -a -f $mnt/iozonetest" + fi log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] jobs/iozone.yaml: add specified block_size parameter
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/jobs/iozone.yaml b/jobs/iozone.yaml index f281186b..1e2239e5 100644 --- a/jobs/iozone.yaml +++ b/jobs/iozone.yaml @@ -2,6 +2,13 @@ suite: iozone testcase: iozone category: benchmark +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + disk: 2HDD fs: -- 2.23.0

1 0

[PATCH v7 compass-ci 5/6] container/send-internet-mail: start
by Luan Shengde 26 Oct '20

26 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/start | 39 ++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/start diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start new file mode 100755 index 0000000..483f01a --- /dev/null +++ b/container/send-internet-mail/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'send-internet-mail' + +names = Set.new %w[ + ROBOT_EMAIL_ADDRESS + ROBOT_EMAIL_PASSWORD + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %W[ + docker run + --restart=always + --name=send-internet-mail + -d +] + env + %W[ + -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} + -v /etc/localtime:/etc/localtime:ro + -w #{defaults['CCI_SRC']}/container/send-internet-mail + --log-driver json-file + send-internet-mail +] + +cmd += ['ruby', 'send-internet-mail.rb'] + +system(*cmd) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: ignore os-nfs container
by Liu Yinsi 26 Oct '20

26 Oct '20

os-nfs service no need to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 26778a4..b3e5374 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,8 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit + [ "$container_name" == 'sub-fluentd' ] || \ + [ "$container_name" == 'os-nfs' ] && exit if [ -z "$start_only" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH compass-ci] container: docker run delete fluentd configration
by Wu Zhende 26 Oct '20

26 Oct '20

[Why] --log-driver has been configured in the global Docker. By default, each docker uses fluentd to collect logs and does not need to be configured separately. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/extract-stats/start | 7 ------- container/scheduler-https/start | 6 ------ container/scheduler/start | 6 ------ container/web-backend-nginx/start | 5 ----- container/web-backend/start | 5 ----- 5 files changed, 29 deletions(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index e5ee777..74cf917 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,6 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - FLUENTD_HOST - FLUENTD_PORT MAIL_HOST MAIL_PORT LAB @@ -24,8 +22,6 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' SRV_HTTP_HOST = defaults['SRV_HTTP_HOST'] || Open3.capture3('curl ip.sb')[0].chomp SRV_HTTP_PORT = defaults['SRV_HTTP_PORT'] || '11300' docker_rm "extract-stats" @@ -44,11 +40,8 @@ cmd = %w[ -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v /srv/result:/result - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=extract-stats extract-stats ] diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 6b06769..38c4892 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -19,8 +19,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST ] @@ -34,8 +32,6 @@ SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -59,8 +55,6 @@ cmd = %W[ -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro -v /etc/compass-ci/ca:/ca - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} diff --git a/container/scheduler/start b/container/scheduler/start index 305916e..e0d70a8 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -17,8 +17,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST GIT_SERVER @@ -42,8 +40,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -71,8 +67,6 @@ cmd = %W[ -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_PORT} diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index a28101d..4da4c8b 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -3,8 +3,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend-nginx @@ -17,11 +15,8 @@ cmd=( -v /etc/ssl/certs:/opt/cert - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend-nginx web-backend-nginx ) diff --git a/container/web-backend/start b/container/web-backend/start index 5fb493c..941fb11 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -5,8 +5,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci [[ $LKP_SRC ]] || LKP_SRC=/c/lkp-tests [[ $UPSTREAM_REPOS_PATH ]] || UPSTREAM_REPOS_PATH=/c/upstream-repos -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend @@ -27,11 +25,8 @@ cmd=( -v $LKP_SRC:/c/lkp-tests -v $UPSTREAM_REPOS_PATH:/c/upstream-repos - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend debian:web-backend ) -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: fix containers auto resume failed after reboot
by Liu Yinsi 26 Oct '20

26 Oct '20

call 4-docker/buildall to fix container auto resume depends issues. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 26 +++++++++++++++++--------- sparrow/7-systemd/cci-network | 9 +++++---- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..26778a4 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -3,6 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. CONTAINER_PATH="$CCI_SRC/container" +export start_only=$1 build_depends() { @@ -30,20 +31,27 @@ do_one() ( cd $container [ "$container_name" == 'sub-fluentd' ] && exit - [ -x build ] && ./build - [ -x install ] && ./install + if [ -z "$start_only" ]; then + [ -x build ] && ./build + [ -x install ] && ./install + fi [ -x first-run ] && ./first-run [ -x start ] && ./start ) lockfile-remove --lock-name "$container_name".lock } -tmpdir=$(mktemp -d) +buildall() +{ + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - build_depends $dir -done + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir + done + + wait + rm -fr $tmpdir +} -wait -rm -fr $tmpdir +buildall diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 4aadcdc..0d21a2d 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -4,13 +4,14 @@ #this file used in our own systemd unit file: cci-network.service. #contents as follows will be automatically executed after system reboot. -[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci +. $CCI_SRC/container/defconfig.sh + +# export ENV +. /etc/profile.d/compass.sh $CCI_SRC/sparrow/2-network/br0 $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs -# --restart=always option is not absolutely reliable -containers=($(docker ps -a |grep -v NAMES |awk '{print $NF}')) -[ -n "$containers" ] && docker start "${containers[@]}" >/dev/null 2>&1 +$CCI_SRC/sparrow/4-docker/buildall start_only -- 2.23.0

1 0

[PATCH compass-ci] sparrow: fix containers auto resume failed after reboot
by Liu Yinsi 26 Oct '20

26 Oct '20

call 4-docker/buildall to fix container auto resume depends issues. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 26 +++++++++++++++++--------- sparrow/7-systemd/cci-network | 9 +++++---- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..26778a4 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -3,6 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. CONTAINER_PATH="$CCI_SRC/container" +export start_only=$1 build_depends() { @@ -30,20 +31,27 @@ do_one() ( cd $container [ "$container_name" == 'sub-fluentd' ] && exit - [ -x build ] && ./build - [ -x install ] && ./install + if [ -z "$start_only" ]; then + [ -x build ] && ./build + [ -x install ] && ./install + fi [ -x first-run ] && ./first-run [ -x start ] && ./start ) lockfile-remove --lock-name "$container_name".lock } -tmpdir=$(mktemp -d) +buildall() +{ + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - build_depends $dir -done + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir + done + + wait + rm -fr $tmpdir +} -wait -rm -fr $tmpdir +buildall diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 4aadcdc..0d21a2d 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -4,13 +4,14 @@ #this file used in our own systemd unit file: cci-network.service. #contents as follows will be automatically executed after system reboot. -[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci +. $CCI_SRC/container/defconfig.sh + +# export ENV +. /etc/profile.d/compass.sh $CCI_SRC/sparrow/2-network/br0 $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs -# --restart=always option is not absolutely reliable -containers=($(docker ps -a |grep -v NAMES |awk '{print $NF}')) -[ -n "$containers" ] && docker start "${containers[@]}" >/dev/null 2>&1 +$CCI_SRC/sparrow/4-docker/buildall start_only -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: add stop reason
by Wu Zhende 26 Oct '20

26 Oct '20

[Why] When the client disconnects from the ws, the reason is not printed. Like this: connect to ws://localhost:11310/filter ............. connection closed: Add stop reason, and it will be like: connect to ws://localhost:11310/filter .......... connection closed: timeout or connection closed: normal Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 9ce1d475..c20794e8 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -27,6 +27,7 @@ class Monitor load_default @result = [] @stop_query = {} + @reason = nil end def load_default @@ -87,16 +88,12 @@ class Monitor ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" end - def stop_em(web_socket) - web_socket.close - EM.stop - end - - def stop(data, web_socket) + def stop(data, web_socket, code = 1000, reason = 'normal') @stop_query.each do |key, value| return false unless data[key] == value end - stop_em(web_socket) + @reason = reason + web_socket.close(code, reason) end def run(timeout = nil) @@ -111,7 +108,8 @@ class Monitor if timeout && timeout != 0 EM.add_timer(timeout) do @exit_status_code = 1 - stop_em(ws) + @reason = 'timeout' + ws.close(1000, @reason) end end @@ -131,7 +129,9 @@ class Monitor end ws.on :close do |event| - puts "connection closed: #{event.reason}" + reason = event.reason || @reason + puts "connection closed: #{reason}" + EM.stop end end return @exit_status_code -- 2.23.0

2 1

[PATCH lkp-tests 1/2] fix(distro/depends): delete unnecessary depends
by Wang Yong 26 Oct '20

26 Oct '20

[Why] ruby-full contains ruby-dev $ apt depends ruby-full ruby-full Depends: ruby Depends: ruby-dev Depends: ri [How] delete this package from depend file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-deploy | 1 - 1 file changed, 1 deletion(-) diff --git a/distro/depends/cci-deploy b/distro/depends/cci-deploy index dac682505..c797c1eea 100644 --- a/distro/depends/cci-deploy +++ b/distro/depends/cci-deploy @@ -4,7 +4,6 @@ g++ make docker.io ruby-full -ruby-dev bridge-utils qemu dmidecode -- 2.23.0

2 2

[PATCH compass-ci] 0-package/common: change the value of fluentd port
by Wu Zhende 26 Oct '20

26 Oct '20

[Why] Prior to this, fluentd used port 24224 to collection docker logs. But now, fluentd change to sub-fluentd, and uses 24225. master-fluentd uses port 24224 to get docker logs from sub-fluentd. So docker should forward logs to sub-fluentd and uses 24225 port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 31588f4..523438c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -31,7 +31,7 @@ cat > /etc/docker/daemon.json <<EOF { "log-driver": "fluentd", "log-opts":{ - "fluentd-address": "localhost:24224", + "fluentd-address": "localhost:24225", "fluentd-async-connect": "true", "tag": "{{.Name}}" }, -- 2.23.0

1 0

[PATCH lkp-tests] fix(adaptation/*): update adaptation for several OS
by Wang Yong 26 Oct '20

26 Oct '20

[Why] some packages in other OS have different name or some Debian packages no need add mapping in red-hat related OS. Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/centos | 10 ++++++++-- distro/adaptation/fedora | 6 +++++- distro/adaptation/kylin | 10 ++++++++++ distro/adaptation/openeuler | 9 +++++++++ distro/adaptation/oracle | 4 +++- 5 files changed, 35 insertions(+), 4 deletions(-) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index f92cee2ff..e7fabf36d 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -8,7 +8,6 @@ build-essential: make automake gcc gcc-c++ kernel-devel rpm-build cgroup-bin: libcgroup clang-6.0: clang cmake: cmake cmake3 -dbench=4.0-2: dbench dnsutils: bind-utils dpkg-dev: dpkg-devel e2fslibs-dev: e2fsprogs-devel @@ -297,7 +296,6 @@ libtasn1-6: libtasn1 libtimedate-perl: perl-TimeDate libtirpc-common: libtirpc libtirpc3: libtirpc -network-manager: network-manager-applet libv4l-dev: libv4l-devel libvirt-clients: libvirt-client libvirt-daemon-system: libvirt-daemon @@ -328,3 +326,11 @@ dh-python: dh-strip-nondeterminism: dh-systemd: guile-2.0: guile +libpython2.7: python-libs +network-manager: NetworkManager +init-system-helpers: +javahelper: +junit4: junit +liboath-dev: liboath-devel +lsb-release: redhat-lsb +linux-base: diff --git a/distro/adaptation/fedora b/distro/adaptation/fedora index dd274d0f1..38b7e9dc4 100644 --- a/distro/adaptation/fedora +++ b/distro/adaptation/fedora @@ -5,7 +5,6 @@ binutils-dev: binutils-devel btrfs-tools: btrfs-progs build-essential: make automake gcc gcc-c++ kernel-devel rpm-build cgroup-bin: libcgroup -dbench=4.0-2: dbench dpkg-dev: dpkg-devel freeglut3: freeglut fsmark: fs_mark @@ -158,3 +157,8 @@ libslang2-dev: slang-devel python-dev: python3-devel systemtap-sdt-dev: systemtap-sdt-devel initramfs-tools: +init-system-helpers: +network-manager: NetworkManager +javahelper: +junit4: junit +linux-base: diff --git a/distro/adaptation/kylin b/distro/adaptation/kylin index c60a7f3ee..f143fde96 100644 --- a/distro/adaptation/kylin +++ b/distro/adaptation/kylin @@ -137,3 +137,13 @@ uuid-runtime: uuid xfslibs-dev: xfsprogs-devel xutils-dev: imake xorg-x11-util-macros zlib1g-dev: zlib-devel +libpython2.7: +network-manager: NetworkManager +wpasupplicant: wpa_supplicant +gdbserver: gdb-gdbserver +init-system-helpers: +javahelper: +lsb-release: +openjdk: java-1.8.0-openjdk +ruby-full: ruby ruby-devel +linux-base: diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index ca8688b5a..e72f34fd2 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -284,3 +284,12 @@ guile-2.0: guile libmysqlclient-dev: mysql-devel libipc-run-perl: libklibc-dev: +libpython2.7: +network-manager: NetworkManager +gdbserver: gdb-gdbserver +init-system-helpers: +javahelper: +lsb-release: +openjdk: java-1.8.0-openjdk +ruby-full: ruby ruby-devel rubygem-rdoc +linux-base: diff --git a/distro/adaptation/oracle b/distro/adaptation/oracle index f7fcfaa84..af8415ec3 100644 --- a/distro/adaptation/oracle +++ b/distro/adaptation/oracle @@ -5,7 +5,6 @@ binutils-dev: binutils-devel btrfs-tools: btrfs-progs build-essential: make automake gcc gcc-c++ kernel-uek-devel rpm-build cgroup-bin: libcgroup -dbench=4.0-2: dbench dpkg-dev: dpkg-devel f2fs-tools: freeglut3: freeglut @@ -133,3 +132,6 @@ xutils-dev: imake xz-utils: xz zlib1g-dev: zlib-devel initramfs-tools: +init-system-helpers: +javahelper: +linux-base: -- 2.23.0

1 0

[PATCH lkp-tests 2/2] fix(distro/depends): move i386 depends to x86_64
by Wang Yong 26 Oct '20

26 Oct '20

[Why] i386 depends are not suitable for aarch64 [How] move these packages to x86_64 file for further support: commit 5b5f5d2b2e1948555dd01c12dc31e77442a542b8 depends: auto load $program.$(arch) Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/kernel-selftests | 22 ---------------------- distro/depends/kernel-selftests.x86_64 | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 22 deletions(-) create mode 100644 distro/depends/kernel-selftests.x86_64 diff --git a/distro/depends/kernel-selftests b/distro/depends/kernel-selftests index 3d08c85da..b02e37e22 100644 --- a/distro/depends/kernel-selftests +++ b/distro/depends/kernel-selftests @@ -2,21 +2,8 @@ gcc-8 gcc-8-multilib libc6-dev libc-bin -libc6-dev-i386 (x86_64) -libc6-dev-x32 libc6/unstable -lib32gcc-8-dev -libx32gcc-8-dev -libc6-i386 libc6-dev -libc6-x32 -libx32gcc1 -libx32gomp1 -libx32itm1 -libx32atomic1 -libx32asan5 -libx32ubsan1 -libx32quadmath0 rsync make libcap-ng-dev @@ -55,12 +42,3 @@ sendip ipvsadm libmnl-dev libkeyutils-dev -lib64gcc1 (i386) -lib64gomp1 (i386) -lib64itm1 (i386) -lib64atomic1 (i386) -lib64asan3 (i386) -libc6-amd64/unstable (i386) -libc6-dev-amd64/unstable (i386) -lib64gcc-8-dev (i386) -libx32gcc-8-dev (i386) diff --git a/distro/depends/kernel-selftests.x86_64 b/distro/depends/kernel-selftests.x86_64 new file mode 100644 index 000000000..9491ac3b5 --- /dev/null +++ b/distro/depends/kernel-selftests.x86_64 @@ -0,0 +1,23 @@ +libc6-dev-i386 (x86_64) +libc6-dev-x32 +libc6-i386 +libc6-x32 +libx32gcc1 +libx32gomp1 +libx32itm1 +libx32atomic1 +libx32asan5 +libx32ubsan1 +libx32quadmath0 +lib64gcc1 (i386) +lib64gomp1 (i386) +lib64itm1 (i386) +lib64atomic1 (i386) +lib64asan3 (i386) +libc6-amd64/unstable (i386) +libc6-dev-amd64/unstable (i386) +lib64gcc-8-dev (i386) +libx32gcc-8-dev (i386) +lib32gcc-8-dev +libx32gcc-8-dev + -- 2.23.0

1 0

[PATCH v7 compass-ci 6/6] container/send-internet-mail: send-mail.md
by Luan Shengde 26 Oct '20

26 Oct '20

documentation for send mail service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/send-mail.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 doc/send-mail.md diff --git a/doc/send-mail.md b/doc/send-mail.md new file mode 100644 index 0000000..63716d7 --- /dev/null +++ b/doc/send-mail.md @@ -0,0 +1,58 @@ + +# send mail + +## purpose + +The service is used to send mails with formatted mail data +- send intranet mail +- send internet mail + +## send mail port +- send intranet mail: 11311 +- send internet mail: 49000 + +## send mail host +- host the service running on + +## data format +- send_mail_yaml + - data format + data='{ + "subject": "email subject", + "to": "email_to_addr", + "body": "email message" + }' + or + data=" + subject: email subject + to: email_to_addr + body: email message + " + +- send_mail_text + - data format + data=" + To: email_to_addr + Subject: email_subject + + mail_msg_line1 + mail_msg_line2 + ... + " + +## usage: +- send_mail_yaml + curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_yaml' -d "$data" +- send_mail_text + curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "$data" + +## example + + data = " + To: test_email(a)163.com + Subject: test mail 10 + + test msg 1010 + " + + %x(curl -XPOST 'localhost:49000/send_mail_text' -d "#{data}") -- 2.23.0

1 0

[PATCH v7 compass-ci 4/6] container/send-internet-mail: build
by Luan Shengde 26 Oct '20

26 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/send-internet-mail/build diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build new file mode 100755 index 0000000..078adaf --- /dev/null +++ b/container/send-internet-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t send-internet-mail:latest . -- 2.23.0

1 0

[PATCH v7 compass-ci 3/6] container/send-internet-mail: Dockerfile
by Luan Shengde 26 Oct '20

26 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/Dockerfile | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 container/send-internet-mail/Dockerfile diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile new file mode 100644 index 0000000..d151bda --- /dev/null +++ b/container/send-internet-mail/Dockerfile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian +MAINTAINER shdluan(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-mail ruby-json ruby-sinatra -- 2.23.0

1 0

[PATCH v7 compass-ci 2/6] container/send-internet-mail: send-internet-mail.rb
by Luan Shengde 26 Oct '20

26 Oct '20

setup smtp config for internet email server invoke mail-post to hand the send mail request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../send-internet-mail/send-internet-mail.rb | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 container/send-internet-mail/send-internet-mail.rb diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb new file mode 100644 index 0000000..3b1d159 --- /dev/null +++ b/container/send-internet-mail/send-internet-mail.rb @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'mail' +require 'sinatra' + +require "#{ENV['CCI_SRC']}/lib/mail-post" + +smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: ENV['ROBOT_EMAIL_ADDRESS'], + password: ENV['ROBOT_EMAIL_PASSWORD'], + openssl_verify_mode: 'none', + enable_starttls_auto: true +} + +Mail.defaults { delivery_method :smtp, smtp } -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci October 2020