January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd initrd [why] initrd is a file, no path involved, can use directly, so remove $initrd, set_initrd() also unnecessary if directly use initrd. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c58d94..1e3f397 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -69,18 +69,13 @@ check_option_value() [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 if [ -n "$initrds" ]; then - cat $initrds > $initrd + cat $initrds > initrd else log_error "The current initrds is null." exit 1 fi } -set_initrd() -{ - initrd=initrd -} - set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd @@ -137,7 +132,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -196,7 +191,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 2

[PATCH v3 lkp-tests] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 02 Feb '21

02 Feb '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/common | 13 +++++++++++++ tests/cci-depends | 9 ++------- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/distro/common b/distro/common index 1fd7df4e8..da2a57a79 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,16 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + + cd "$pack_to" + [ -f "${BM_NAME}".cgz ] || return + rm "${BM_NAME}".cgz + date=$(ls -F | grep -v "@" | grep -Eo "[0-9]{8}") + mv "${BM_NAME}"_"$date".cgz "${benchmark}"_"$date".cgz + + ln -s "${benchmark}"_"$date".cgz "${benchmark}".cgz +} diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 1

[PATCH compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'y_params'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 39 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..6f85f35 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "x_params": [$date1, ..., $date_today], +# "y_params": [$times_1, ..., $times_today] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..1c29c88 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,42 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'x_params' => [], + 'y_params' => [] + } + dev = 0 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['x_params'][dev] = day_s + echart['y_params'][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci] git-mirror: fix build error
by Yu Chuan 01 Feb '21

01 Feb '21

[Why] After `apt add ruby-dev` is executed, the `gem` command can be available. [Error Msg] Step 6/7 : RUN apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add--no-cache git && apk add ruby-dev make gcc g++ openssh ---> Running in 4c1f8708c072 fetch http://mirrors.huaweicloud.com/alpine/v3.8/main/aarch64/APKINDEX.tar.gz fetch http://mirrors.huaweicloud.com/alpine/v3.8/community/aarch64/APKINDEX.tar.gz v3.8.5-65-g097f0358a9 [http://mirrors.huaweicloud.com/alpine/v3.8/main] v3.8.5-37-gf06ffe835a [http://mirrors.huaweicloud.com/alpine/v3.8/community] OK: 9385 distinct packages available /bin/sh: gem: not found The command '/bin/sh -c apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add --no-cache git && apk add ruby-dev make gcc g++ openssh' returned a non-zero code: 127 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/git-mirror/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 2eac95c54ca1..18613eddf30e 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,10 +10,10 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ - apk add ruby-dev make gcc g++ openssh + apk add ruby-dev make gcc g++ openssh && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN umask 002 && \ gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

2 3

[PATCH compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..e4c2cbd --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ +RUN gem sources --remove https://rubygems.org/ +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

2 4

[PATCH lab-z9] add host-file of a35 a37 a39
by Zhang Dewan 01 Feb '21

01 Feb '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a35 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a37 | 27 +++++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 +++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s64p-256g--a37 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..5d9d500 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,19 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a37 b/hosts/taishan200-2280-2s64p-256g--a37 new file mode 100644 index 0000000..657707a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a37 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399c802ef2d +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part3 +mac_addr: + - 84:46:fe:89:b9:66 + - 84:46:fe:89:b9:67 + - 84:46:fe:89:b9:68 + - 84:46:fe:89:b9:69 + - ac:b3:b5:1e:d5:eb + - ac:b3:b5:1e:d5:ec + - ac:b3:b5:1e:d5:ed + - ac:b3:b5:1e:d5:ee + - ac:b3:b5:1e:d6:3b + - ac:b3:b5:1e:d6:3c + - ac:b3:b5:1e:d6:3d + - ac:b3:b5:1e:d6:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.2 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

2 4

[PATCH lkp-tests] lib/job.rb: delete fields that need to be reloaded for reproduction task
by Li Ping 01 Feb '21

01 Feb '21

[why] 1. load job from result's job yaml has key "my_email", like my_email: compass-ci-robot(a)qq.com my_name: compass-ci-robot lab: z9 2. load job from result's job yaml has key "SCHED_HOST" which from /etc/compass-ci/defaults/*.yaml, like SCHED_HOST: 172.168.131.2 SCHED_PORT: 3000 each of these fields needs to be reset when submit job.yaml for reprodution task. however, these fields already exist in yaml file, can not to be reload, so delete these fields when submit reproduction job. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 640b25a5f..867c3d193 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -258,6 +258,13 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + if hash.has_key?('id') + hash.delete_if { |key, _| key == "SCHED_HOST" } + hash.delete_if { |key, _| key == "SCHED_PORT" } + hash.delete_if { |key, _| key == "my_name" } + hash.delete_if { |key, _| key == "my_email" } + hash.delete_if { |key, _| key == "lab" } + end revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? -- 2.23.0

3 4

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4

[PATCH compass-ci 4/4] sparrow/5-build: pack lkp-tests
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for user locally deploy compass-ci, submit job will check lkp-$arch.cgz exist, so call container lkp-initrd/run in sparrow script, it is convenient and does not need users to operate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources index 48a740b..cea3b00 100755 --- a/sparrow/5-build/sources +++ b/sparrow/5-build/sources @@ -3,3 +3,5 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. $CCI_SRC/sbin/download-rootfs + +$CCI_SRC/container/lkp-initrd/run -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2