- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/dump_stat.rb: fix name error
by Lu Weitao 26 Mar '21

26 Mar '21

[Why] /home/baijing/compass-ci/lib/dump_stat.rb:171:in `number?': undefined local variable or method `monitor' for DumpStat:Module (NameError) Did you mean? @monitor Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 1c0695e..de0168e 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -55,7 +55,7 @@ module DumpStat if value.is_a?(String) value = check_string_value(k, value, @monitor) next unless value - return nil unless number?(value, @invalid_records) + return nil unless number?(value, @invalid_records, @record_index, @monitor) value = value.index('.') ? value.to_f : value.to_i elsif value.is_a?(Array) @@ -64,7 +64,7 @@ module DumpStat value[i] = check_string_value(k, value[i], @monitor) next unless value[i] - return nil unless number?(value[i], @invalid_records) + return nil unless number?(value[i], @invalid_records, @record_index, @monitor) value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i valid_stats_verification(k, value[i]) @@ -165,7 +165,7 @@ def check_string_value(key, value, monitor) end # only number is valid -def number?(value, invalid_records) +def number?(value, invalid_records, record_index, monitor) unless value.numeric? invalid_records.push record_index warn_stat "invalid stats value: #{value}", monitor -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/scheduler: fix library libgc.so.1: No such file or directory
by Cao Xueliang 26 Mar '21

26 Mar '21

use the compile env as the runtime env Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/Dockerfile | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 7fb8e35..229a9c7 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories @@ -12,10 +12,6 @@ RUN apk add --no-cache 'ruby-dev' \ RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install rest-client && \ - gem install activesupport && \ - gem install git && \ - gem install json && \ gem install yaml -- 2.23.0

1 1

[PATCH compass-ci 2/3] container/crystal-shards: update crystal version from 0.33 to 0.35
by Cao Xueliang 26 Mar '21

26 Mar '21

add shard-amqp.yml to shard.yml, delete shard-amqp.yml add crystal-etcd to shard.yml Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-shards/Dockerfile | 3 --- container/crystal-shards/shard-amqp.yml | 7 ------- container/crystal-shards/shard.yml | 9 ++++++--- 3 files changed, 6 insertions(+), 13 deletions(-) delete mode 100644 container/crystal-shards/shard-amqp.yml diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 15fe523..d3e321f 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -17,8 +17,5 @@ sed -i '99s/arguments\[:id]/arguments\[:id]?/' /usr/share/crystal/app/lib/elast sed -i 's/, Utils.__listify(arguments\[:q].as(String))/ /' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr;\ sed -i '205a \ params.clear' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr; -COPY shard-amqp.yml /usr/share/crystal/app/shard.yml -RUN shards - CMD ["bash"] diff --git a/container/crystal-shards/shard-amqp.yml b/container/crystal-shards/shard-amqp.yml deleted file mode 100644 index aea87cc..0000000 --- a/container/crystal-shards/shard-amqp.yml +++ /dev/null @@ -1,7 +0,0 @@ -name: monitor -version: 0.1.0 - -dependencies: - amqp-client: - github: cloudamqp/amqp-client.cr - version: 0.5.14 diff --git a/container/crystal-shards/shard.yml b/container/crystal-shards/shard.yml index ca8f72e..3466bc5 100644 --- a/container/crystal-shards/shard.yml +++ b/container/crystal-shards/shard.yml @@ -8,7 +8,7 @@ targets: scheduler: main: src/scheduler.cr -crystal: 0.33.0 +crystal: 0.35.0 license: Mulan PSL v2 @@ -17,10 +17,8 @@ dependencies: github: kemalcr/kemal redis: github: stefanwille/crystal-redis - version: ~> 2.5.3 elasticsearch-crystal: github: paktek123/elasticsearch-crystal - version: ~> 0.14 any_merge: github: icyleaf/any_merge branch: master @@ -32,3 +30,8 @@ dependencies: github: z64/rate_limiter singleton: github: waterlink/singleton.cr + etcd: + github: PythonEngineer007/crystal-etcd + branch: master + amqp-client: + github: cloudamqp/amqp-client.cr -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/crystal-base: update alpine version from 3.11 to 3.13
by Cao Xueliang 26 Mar '21

26 Mar '21

need install llvm10 in alpine:3.13. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-base/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/crystal-base/Dockerfile b/container/crystal-base/Dockerfile index 03fe5e6..633a2ec 100644 --- a/container/crystal-base/Dockerfile +++ b/container/crystal-base/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:3.13 MAINTAINER chief <taxcom(a)tom.com> @@ -10,7 +10,7 @@ ONBUILD RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.hu ONBUILD RUN apk update ONBUILD RUN apk add bash zsh git rsync curl crystal shards tzdata sudo \ yaml-dev openssl-dev ncurses-static g++ make cmake \ - zlib-dev zlib-static openssl-libs-static llvm5 llvm5-dev \ - llvm5-libs llvm5-static musl-dev xz-dev libxml2-dev \ + zlib-dev zlib-static openssl-libs-static llvm10 llvm10-dev \ + llvm10-libs llvm10-static musl-dev xz-dev libxml2-dev \ vim ruby-dev -- 2.23.0

1 0

[PATCH compass-ci] container/result-webdav: add /srv/rpm share dir
by Cui Yili 26 Mar '21

26 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index 2852f24..993f24c 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/rpm/:/srv/rpm/ -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) -- 2.23.0

1 0

[PATCH lab-z9] add testboxes host-file
by Zhang Dewan 26 Mar '21

26 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a63 | 6 +++--- hosts/taishan200-2280-2s48p-256g--a64 | 6 +++--- hosts/taishan200-2280-2s48p-256g--a65 | 6 +++--- hosts/taishan200-2280-2s64p-256g--a110 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a112 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a113 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a114 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a116 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a118 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a119 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a124 | 22 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 ++++++++++++++++++++++++++ 12 files changed, 191 insertions(+), 9 deletions(-) create mode 100644 hosts/taishan200-2280-2s64p-256g--a110 create mode 100644 hosts/taishan200-2280-2s64p-256g--a112 create mode 100644 hosts/taishan200-2280-2s64p-256g--a113 create mode 100644 hosts/taishan200-2280-2s64p-256g--a114 create mode 100644 hosts/taishan200-2280-2s64p-256g--a116 create mode 100644 hosts/taishan200-2280-2s64p-256g--a118 create mode 100644 hosts/taishan200-2280-2s64p-256g--a119 create mode 100644 hosts/taishan200-2280-2s64p-256g--a124 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 index a8e6688..0b3c938 100644 --- a/hosts/taishan200-2280-2s48p-256g--a63 +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:e5:49 - 44:67:47:c9:e5:4a - 44:67:47:c9:e5:4b +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 index f8aa111..7cd0c48 100644 --- a/hosts/taishan200-2280-2s48p-256g--a64 +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:ea:b7 - 44:67:47:c9:ea:b8 - 44:67:47:c9:ea:b9 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.5 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 index 341303c..88441cc 100644 --- a/hosts/taishan200-2280-2s48p-256g--a65 +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:e9:95 - 44:67:47:c9:e9:96 - 44:67:47:c9:e9:97 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.6 diff --git a/hosts/taishan200-2280-2s64p-256g--a110 b/hosts/taishan200-2280-2s64p-256g--a110 new file mode 100644 index 0000000..50625bc --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a110 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6079 + - /dev/disk/by-id/scsi-350000399886b69bd +mac_addr: + - 44:67:47:e9:79:3d + - 44:67:47:e9:79:3e + - 44:67:47:e9:79:3f + - 44:67:47:e9:79:40 + - 44:a1:91:9f:9b:8a + - 44:a1:91:9f:9b:8b + - 44:a1:91:9f:9b:8c + - 44:a1:91:9f:9b:8d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.3 diff --git a/hosts/taishan200-2280-2s64p-256g--a112 b/hosts/taishan200-2280-2s64p-256g--a112 new file mode 100644 index 0000000..30c0f80 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a112 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399888173c1 + - /dev/disk/by-id/scsi-350000399888169c9 +mac_addr: + - 44:a1:91:9d:f3:3a + - 44:a1:91:9d:f3:3b + - 44:a1:91:9d:f3:3c + - 44:a1:91:9d:f3:3d + - 44:a1:91:e5:85:11 + - 44:a1:91:e5:85:12 + - 44:a1:91:e5:85:13 + - 44:a1:91:e5:85:14 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a113 b/hosts/taishan200-2280-2s64p-256g--a113 new file mode 100644 index 0000000..826b18d --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a113 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988700409 + - /dev/disk/by-id/scsi-350000399886bfb25 +mac_addr: + - 44:67:47:85:ed:b7 + - 44:67:47:85:ed:b8 + - 44:67:47:85:ed:b9 + - 44:67:47:85:ed:ba + - cc:05:77:ee:b3:1f + - cc:05:77:ee:b3:20 + - cc:05:77:ee:b3:21 + - cc:05:77:ee:b3:22 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.6 diff --git a/hosts/taishan200-2280-2s64p-256g--a114 b/hosts/taishan200-2280-2s64p-256g--a114 new file mode 100644 index 0000000..f3a4d68 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a114 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399785bf105 + - /dev/disk/by-id/scsi-350000399785bf189 +mac_addr: + - 44:a1:91:9d:d5:44 + - 44:a1:91:9d:d5:45 + - 44:a1:91:9d:d5:46 + - 44:a1:91:9d:d5:47 + - 44:a1:91:e5:6f:01 + - 44:a1:91:e5:6f:02 + - 44:a1:91:e5:6f:03 + - 44:a1:91:e5:6f:04 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.7 diff --git a/hosts/taishan200-2280-2s64p-256g--a116 b/hosts/taishan200-2280-2s64p-256g--a116 new file mode 100644 index 0000000..5ca6cd4 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a116 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6fad + - /dev/disk/by-id/scsi-350000399886b6ba9 +mac_addr: + - 44:67:47:85:f0:af + - 44:67:47:85:f0:b0 + - 44:67:47:85:f0:b1 + - 44:67:47:85:f0:b2 + - cc:05:77:ee:b0:ef + - cc:05:77:ee:b0:f0 + - cc:05:77:ee:b0:f1 + - cc:05:77:ee:b0:f2 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.9 diff --git a/hosts/taishan200-2280-2s64p-256g--a118 b/hosts/taishan200-2280-2s64p-256g--a118 new file mode 100644 index 0000000..9a93e03 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a118 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988707e91 + - /dev/disk/by-id/scsi-350000399887080e5 +mac_addr: + - 44:67:47:e9:78:cf + - 44:67:47:e9:78:d0 + - 44:67:47:e9:78:d1 + - 44:67:47:e9:78:d2 + - 44:a1:91:9f:dd:aa + - 44:a1:91:9f:dd:ab + - 44:a1:91:9f:dd:ac + - 44:a1:91:9f:dd:ad +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.11 diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 new file mode 100644 index 0000000..57da863 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6be9 + - /dev/disk/by-id/scsi-350000399886b5dbd +mac_addr: + - 44:67:47:e9:79:65 + - 44:67:47:e9:79:66 + - 44:67:47:e9:79:67 + - 44:67:47:e9:79:68 + - 44:a1:91:9f:9b:4a + - 44:a1:91:9f:9b:4b + - 44:a1:91:9f:9b:4c + - 44:a1:91:9f:9b:4d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a124 b/hosts/taishan200-2280-2s64p-256g--a124 new file mode 100644 index 0000000..7e7bb42 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a124 @@ -0,0 +1,22 @@ +memory: 256G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44344 +mac_addr: + - 44:67:47:e9:76:9f + - 44:67:47:e9:76:a0 + - 44:67:47:e9:76:a1 + - 44:67:47:e9:76:a2 + - ac:b3:b5:1e:d5:3b + - ac:b3:b5:1e:d5:3c + - ac:b3:b5:1e:d5:3d + - ac:b3:b5:1e:d5:3e + - ac:b3:b5:1e:d6:6b + - ac:b3:b5:1e:d6:6c + - ac:b3:b5:1e:d6:6d + - ac:b3:b5:1e:d6:6e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: change upload dir
by Wang Yong 26 Mar '21

26 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c1ca223..5767f61 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" + package_dir = ",/rpm/upload/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change upload dir
by Wang Yong 26 Mar '21

26 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 90905b220..7a101b6a0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -11,7 +11,7 @@ [ -n "$upstream_repo" ] || die "upstream_repo is empty" -dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" +dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" from_git() { -- 2.23.0

1 0

[PATCH compass-ci] doc: fix some spell errors
by Li Yuanchao 26 Mar '21

26 Mar '21

blacklight => backlight Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 2 +- doc/manual/test-oss-project.zh.md | 4 ++-- .../{fork_blacklight.png => fork_backlight.png} | Bin 3 files changed, 3 insertions(+), 3 deletions(-) rename doc/pictures/{fork_blacklight.png => fork_backlight.png} (100%) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index a3c0781..6b6d9ca 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -8,7 +8,7 @@ Perform the following steps to add the information of the code repository to be 1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. Run the following command to create a file path named with the repository name and its initial letter: diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index b455c32..413066d 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -6,9 +6,9 @@ 执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 -1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 backlight 仓库（https://github.com/baskerville/backlight ）为例说明。 -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. 执行以下命令，以首字母和仓库名创建文件路径。 diff --git a/doc/pictures/fork_blacklight.png b/doc/pictures/fork_backlight.png similarity index 100% rename from doc/pictures/fork_blacklight.png rename to doc/pictures/fork_backlight.png -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $MAIL_HOST
by Liu Yinsi 25 Mar '21

25 Mar '21

MAIL_HOST => SEND_MAIL_HOST [why] because MAIL_HOST is duplicate variable name, $MAIL_HOST is $SEND_MAIL_HOST. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index bca4ab435..4f1055033 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -87,5 +87,5 @@ send_email() ${subject}_content - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-10001}/send_mail_text" -d "$email_content" + curl -XPOST "${SEND_MAIL_HOST:-$LKP_SERVER}:${SEND_MAIL_PORT:-10001}/send_mail_text" -d "$email_content" } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..d61f11e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,21 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +68,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +105,18 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..f96972e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,22 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${MONITOR_PORT:=20001} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +69,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +106,19 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +MONITOR_PORT: $MONITOR_PORT +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: update SRV_HTTP_RESULT_PORT
by Liu Yinsi 25 Mar '21

25 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- lib/monitor.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/email.sh b/lib/email.sh index 63d5a51e2..bca4ab435 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -8,7 +8,7 @@ Subject: [NOTIFY Compass-CI] Test job failed: $id Dear $my_name: Sorry to inform you that your test job is failed, you can click the following link to view details. - http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… Regards Compass-CI diff --git a/lib/monitor.rb b/lib/monitor.rb index f14e131e3..bf7ef69ba 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -100,9 +100,9 @@ class Monitor @result_roots.each do |res| res.to_s.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + srv_http_result_host = job['SRV_HTTP_RESULT_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_result_port = job['SRV_HTTP_RESULT_PORT'] || '20007' + url = "http://#{srv_http_result_host}:#{srv_http_result_port}#{res}" system "lftp -c mirror #{url} >/dev/null 2>&1" end end -- 2.23.0

1 0

help star
by Wu Fengguang 25 Mar '21

25 Mar '21

Yu Chuan, Wu Zhende is our help star again, thanks Yu Chuan, Wu Zhende! wfg@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-03-25.md 15.2 Yu Chuan 15.1 Wu Zhende 11.2 Cao Xueliang 11.0 Wei Jihui 8.8 Liu Yinsi 7.4 Luan Shengde 7.0 Wang Yong 6.4 Lu Weitao 5.6 Wu Fengguang 5.4 Wang Chenglong 5.1 Zhang Dewan 5.0 Li Yuanchao 4.2 Liu Shaofei 4.1 Lin Jiaxin 2.9 Li Ping 1.8 Du Kaitian 1.6 Xu Xijian 1.2 Bai Jing 0.9 Cui Yili 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 25 Mar '21

25 Mar '21

Wei Jihui is our code star, thanks Wei Jihui! wfg@crystal ~/todo/bin% ./git-count 6 Wei Jihui 5 Li Yuanchao 4 Wu Zhende 4 Yu Chuan 4 Luan Shengde 4 Lin Jiaxin 3 Cao Xueliang 3 Liu Shaofei 3 Liu Yinsi 3 Li Ping 2 Bai Jing 2 Xu Xijian 2 Lu Weitao 2 Cui Yili 2 Wang Chenglong 2 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

review star
by Wu Fengguang 25 Mar '21

25 Mar '21

Luan Shengde is our review star, thanks Luan Shengde! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 9 4 19 9 Wu Fengguang 19 9 18 7 Luan Shengde 49 6 6 6 Wei Jihui 21 5 5 6 Liu Shaofei 39 2 2 5 Cao Xueliang 14 4 4 3 Lin Jiaxin 22 3 5 2 Li Yuanchao 3 0 0 2 Xu Xijian 8 3 5 2 Bai Jing 20 2 2 1 Wang Chenglong 17 3 5 1 Wang Yong 9 1 0 1 Lu Weitao 7 2 3 1 Cui Yili 3 0 0 0 Zhang Dewan 4 1 1 0 Yu Chuan 14 0 0 0 Wu Zhende 11 1 1 0 Li Ping 9 0 0 0 Liu Yinsi

1 0

[PATCH v5 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 25 Mar '21

25 Mar '21

[Example] curl -X GET 'localhost:10002/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "attributes":["job_id","error_id","error_message","link_to_result"], "objects":[ { "job_id": "crystal.1354921", "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…", ate/crystal.1354921" }, { "job_id": "crystal.1354929", ... }, ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 18 +++++++++ src/lib/web_backend.rb | 65 +++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index adb91e0..c7d1ec6 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,4 +170,22 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter": {"group_id":"wcl_ansible-openeuler-03-10"}, +# "attributes": ["job_id", "error_id", "error_message", "link_to_result"], +# "objects": [ +# { +# "job_id": "crystal.1354921", +# "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", +# "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# }, +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 40471e0..7f26b9c 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -613,4 +614,68 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_objects = get_error_objects(params) + { + filter: params, + attributes: ['job_id', 'error_id', 'error_message', 'link_to_result'], + objects: error_objects, + }.to_json +end + +def get_error_objects(filter_items) + error_objs = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_obj = get_error_from_job(job) + error_objs << error_obj unless error_obj.empty? + end + + error_objs +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error_obj = {} + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error_obj['id'] = job['id'] + job_error_obj['error_id'] = error_id + job_error_obj['error_message'] = value + job_error_obj['link_to_result'] = value + end + + job_error_obj end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..b2c1814 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,33 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} + mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +109,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..005e29e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,32 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +108,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/build-pkg: update filtering rule
by Lin Jiaxin 25 Mar '21

25 Mar '21

error_ids that are filtered out: /tmp/lkp/clementine-git/src/Clementine/3rdparty/qsqlite/qsql_sqlite.cpp: In function ‘QSqlError qMakeError(sqlite3*, const QString&, QSqlError::ErrorType, int)’: make: *** [makefile.linux:68: makefile.dependencies] Error 1 go: finding github.com/pingcap/errors v0.11.5-0.20201126102027-b0a155152ca3 Cyclomatic Complexity 8 ./include/trace/events/error_report.h:trace_error_report_end etc. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/build-pkg b/stats/build-pkg index 90f27656..764594b7 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -20,7 +20,7 @@ while (line = STDIN.gets) seqno = '' end next unless in_stderr - next unless line.downcase =~ /error|warning/ + next unless line.downcase =~ /(error|warning):[^:]/ error_ids << build_pkg_error_id(line) end -- 2.23.0

3 2

[PATCH lkp-tests] stats/ansible_test: add module name in error id
by Wang Chenglong 25 Mar '21

25 Mar '21

before: error.Failed-to-install-some-of-the-specified-packages: 1 after: error.[msg].Failed-to-install-some-of-the-specified-packages: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index d6acad1ab..3d3b0b0cb 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -2,18 +2,18 @@ require 'json' -def output_error(error_msg) +def output_error(error_msg, error_key) return if error_msg.nil? if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end end @@ -33,8 +33,8 @@ while (line = STDIN.gets) ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'] - output_error ansible_failed_json['message'] - output_error ansible_failed_json['cmd'] - output_error ansible_failed_json['failures'] + output_error ansible_failed_json['msg'],'msg' + output_error ansible_failed_json['message'],'message' + output_error ansible_failed_json['cmd'],'cmd' + output_error ansible_failed_json['failures'],'failures' end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 25 Mar '21

25 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 67 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..90905b220 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,30 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + init_workspace() { @@ -20,30 +43,43 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + local url="git://${GIT_SERVER}/openeuler/${upstream_repo}" + + git clone "$url" || die "clone git repo ${package_name} failed: $url" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm: ${repo_addr}/${upstream_repo}" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +107,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +[ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 25 Mar '21

25 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 73 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..282d473fd 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,38 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + +check_flow() +{ + if [ -n "$repo_name" ]; then + from_srpm + else + from_git + fi +} init_workspace() { @@ -20,30 +51,41 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + git clone "git://${GIT_SERVER}/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://${GIT_SERVER}/openeuler/${upstream_repo}" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm: ${repo_addr}/${upstream_repo}" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +113,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +check_flow build_rpm upload_rpm_pkg -- 2.23.0

3 6

[PATCH v5 compass-ci 5/5] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..a3c0781 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How Do I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

1 0

[PATCH v5 compass-ci 4/5] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v5 compass-ci 3/5] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 24 Mar '21

24 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/5] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, "chenqun": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..40471e0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,67 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_stats = { 'nr_fail' => 0, 'nr_pass' => 0, 'nr_all' => 0 } + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_stats['nr_fail'] += fail_count + nr_stats['nr_pass'] += pass_count + nr_stats['nr_all'] += all_count + end + nr_stats +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/5] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..adb91e0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail: $nr_fail +# }, +# "chenqun": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail": $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

1 0

[PATCH v4 compass-ci 5/5] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..7b72e2b 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How DO I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

2 2

[PATCH v2 compass-ci] container/srv-http/Dockerfile: build h5ai by builder
by Lu Weitao 24 Mar '21

24 Mar '21

[Why] use node:alpine as a Builder for build h5ai we can reduce install app such: nginx, node, npm to save deploy time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 6564730..b7d0677 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,25 +1,32 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.9 +FROM node:alpine as Builder + +MAINTAINER Lu Weitao <luweitaobe(a)163.com> + +RUN apk add git + +RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai + +WORKDIR /h5ai + +RUN npm install && \ + npm run build + +FROM nginx:alpine ADD root / -RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm git \ - && mkdir /run/nginx/ +RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai -WORKDIR /h5ai +COPY --from=Builder /h5ai/build/_h5ai /srv/_h5ai -# build h5ai package -RUN npm install \ - && npm run build \ - && cp -r ./build/_h5ai /srv \ - && chown -R nginx:nginx /srv/_h5ai +RUN chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http/Dockerfile: build h5ai by builder
by Lu Weitao 24 Mar '21

24 Mar '21

[Why] use node:alpine as a Builder for build h5ai we can reduce install app such: nginx, node, npm to save deploy time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 6564730..81a8a45 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,25 +1,30 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.9 +FROM node:alpine as Builder + +RUN apk add git + +RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai + +WORKDIR /h5ai + +RUN npm install && \ + npm run build + +FROM nginx:alpine ADD root / -RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm git \ - && mkdir /run/nginx/ +RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai -WORKDIR /h5ai +COPY --from=Builder /h5ai/build/_h5ai /srv/_h5ai -# build h5ai package -RUN npm install \ - && npm run build \ - && cp -r ./build/_h5ai /srv \ - && chown -R nginx:nginx /srv/_h5ai +RUN chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] -- 2.23.0

2 2

[PATCH v4 compass-ci 2/5] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, "chenqun": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 68 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..04bccdb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + nr_all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + nr_all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'nr_all' => nr_all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 24 Mar '21

24 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 73 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..17169a31b 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,38 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + +check_flow() +{ + if [ -n "$repo_name" ]; then + from_srpm + else + from_git + fi +} init_workspace() { @@ -20,30 +51,41 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + git clone "git://${GIT_SERVER}/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://${GIT_SERVER}/openeuler/${upstream_repo}" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +113,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +check_flow build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] provides: fix require error
by Wu Zhende 24 Mar '21

24 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use require_relative Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index d3f8406..01ba3d5 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -7,7 +7,7 @@ require 'fileutils' require 'optparse' require 'json' -require '../lib/mq_client' +require_relative "../lib/mq_client" opt = {} options = OptionParser.new do |opts| -- 2.23.0

2 1

[PATCH v3 compass-ci 1/2] service/sshr: change port
by Wu Zhende 24 Mar '21

24 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

1 0

[PATCH v2 compass-ci] container/mail-robot: update port
by Luan Shengde 24 Mar '21

24 Mar '21

services have assigned new port: assign-account: 29999 => 10000 send-internet-mail: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/lib/apply-jumper-account.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 97c3989..798546a 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,7 +12,7 @@ require_relative 'assign-account-fail-eamil' require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' -SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 10001 # assign uuid/account for user # when mail-robot listened new email, and the email's subject diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb index 73d2946..9043309 100755 --- a/container/mail-robot/lib/apply-jumper-account.rb +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -6,7 +6,7 @@ require 'json' JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' -JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +JUMPER_PORT = ENV['JUMPER_PORT'] || 10000 # used to apply account # be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] service/sshr: change port
by Wu Zhende 24 Mar '21

24 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] provides: fix require error
by Wu Zhende 24 Mar '21

24 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use absolute path Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 4 +++- providers/multi-qemu | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 9d6854f..b372a11 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -3,12 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'open-uri' require 'json' require 'set' require 'fileutils' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' diff --git a/providers/multi-qemu b/providers/multi-qemu index d3f8406..70675d1 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -3,11 +3,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'fileutils' require 'optparse' require 'json' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" opt = {} options = OptionParser.new do |opts| -- 2.23.0

2 1

[PATCH v4 compass-ci 4/5] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v4 compass-ci 3/5] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 24 Mar '21

24 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/5] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..adb91e0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail: $nr_fail +# }, +# "chenqun": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail": $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..49c07a7 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming" : { +# "nr_all" : $nr_all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fail +# }, +# "chenqun" : { +# "nr_all" : $nr_all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

2 2

[PATCH compass-ci] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..7b72e2b 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How DO I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/dracut-initrd: add some tools into dracut to enable initrd
by Xu Xijian 24 Mar '21

24 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index b8eb6f5..a57d681 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -7,7 +7,7 @@ export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y --fix-missing --no-install-recommends -qq apt-utils \ nfs-common netbase cifs-utils kmod rsync dracut dracut-network xz-utils \ -dracut-config-generic +dracut-config-generic lvm2 xz-utils systemd-sysv apt-get clean rm -rf /var/lib/apt/lists/* @@ -17,7 +17,8 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/e2fsck /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/00bash/module-setup.sh +tools_00bash="/sbin/e2fsck /sbin/mke2fs /usr/bin/basename /sbin/lvm /sbin/reboot" +sed -i "/install() {/a\ inst $tools_00bash" /usr/lib/dracut/modules.d/00bash/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/dracut-initrd: use rootfs_src to replace root in local mount
by Xu Xijian 24 Mar '21

24 Mar '21

$rootfs_src is what we want to get timestamp and os_info, $rootfs_src is fixed formatted, $root can have several other format. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 240551a..835bdd2 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -4,7 +4,7 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $root)" + timestamp="$(basename $rootfs_src)" os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/job.rb: fix host file not exist
by Li Ping 24 Mar '21

24 Mar '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [how] add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox --- lib/job.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b6..8ac5d7a98 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,14 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) - raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" + ["#{lab_repo}/hosts/#{@job['testbox']}", + "#{LKP_SRC}/hosts/#{@job['testbox']}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end + + raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end def include_files -- 2.23.0

1 1

[PATCH v3 lkp-tests] lib/job.rb: fix host file not exist
by Li Ping 24 Mar '21

24 Mar '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [how] add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox --- lib/job.rb | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b6..064a2b1dd 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,15 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + + ["#{lab_repo}/hosts/#{@job['testbox']}", + "#{LKP_SRC}/hosts/#{@job['testbox']}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: use rootfs_src to replace root in local mount
by Xu Xijian 24 Mar '21

24 Mar '21

$rootfs_src is what we want to get timestamp and os_info, $rootfs_src is fixed formatted, $root can have several other format. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 240551a..835bdd2 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -4,7 +4,7 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $root)" + timestamp="$(basename $rootfs_src)" os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix host file not exist
by Wei Jihui 24 Mar '21

24 Mar '21

add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b..c1df0f5b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,16 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + testbox = @job['testbox'] tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" + + ["#{lab_repo}/hosts/#{testbox}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{testbox}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 1