January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/2] providers/qemu: remove unnecessary variable
by Liu Yinsi 08 Feb '21

08 Feb '21

before: -initrd $initrd after: -initrd initrd [why] initrd is a file, no path involved, can use directly, so remove $initrd, set_initrd() also unnecessary if directly use initrd. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 0c58d94..1e3f397 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -69,18 +69,13 @@ check_option_value() [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 if [ -n "$initrds" ]; then - cat $initrds > $initrd + cat $initrds > initrd else log_error "The current initrds is null." exit 1 fi } -set_initrd() -{ - initrd=initrd -} - set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd @@ -137,7 +132,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -196,7 +191,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 2

[PATCH v3 lkp-tests] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 02 Feb '21

02 Feb '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/common | 13 +++++++++++++ tests/cci-depends | 9 ++------- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/distro/common b/distro/common index 1fd7df4e8..da2a57a79 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,16 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + + cd "$pack_to" + [ -f "${BM_NAME}".cgz ] || return + rm "${BM_NAME}".cgz + date=$(ls -F | grep -v "@" | grep -Eo "[0-9]{8}") + mv "${BM_NAME}"_"$date".cgz "${benchmark}"_"$date".cgz + + ln -s "${benchmark}"_"$date".cgz "${benchmark}".cgz +} diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 1

[PATCH compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'y_params'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 39 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..6f85f35 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "x_params": [$date1, ..., $date_today], +# "y_params": [$times_1, ..., $times_today] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..1c29c88 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,42 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'x_params' => [], + 'y_params' => [] + } + dev = 0 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['x_params'][dev] = day_s + echart['y_params'][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci] git-mirror: fix build error
by Yu Chuan 01 Feb '21

01 Feb '21

[Why] After `apt add ruby-dev` is executed, the `gem` command can be available. [Error Msg] Step 6/7 : RUN apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add--no-cache git && apk add ruby-dev make gcc g++ openssh ---> Running in 4c1f8708c072 fetch http://mirrors.huaweicloud.com/alpine/v3.8/main/aarch64/APKINDEX.tar.gz fetch http://mirrors.huaweicloud.com/alpine/v3.8/community/aarch64/APKINDEX.tar.gz v3.8.5-65-g097f0358a9 [http://mirrors.huaweicloud.com/alpine/v3.8/main] v3.8.5-37-gf06ffe835a [http://mirrors.huaweicloud.com/alpine/v3.8/community] OK: 9385 distinct packages available /bin/sh: gem: not found The command '/bin/sh -c apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add --no-cache git && apk add ruby-dev make gcc g++ openssh' returned a non-zero code: 127 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/git-mirror/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 2eac95c54ca1..18613eddf30e 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,10 +10,10 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ - apk add ruby-dev make gcc g++ openssh + apk add ruby-dev make gcc g++ openssh && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN umask 002 && \ gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

2 3

[PATCH compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..e4c2cbd --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ +RUN gem sources --remove https://rubygems.org/ +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

2 4

[PATCH lab-z9] add host-file of a35 a37 a39
by Zhang Dewan 01 Feb '21

01 Feb '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a35 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a37 | 27 +++++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 +++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s64p-256g--a37 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..5d9d500 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,19 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a37 b/hosts/taishan200-2280-2s64p-256g--a37 new file mode 100644 index 0000000..657707a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a37 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399c802ef2d +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part3 +mac_addr: + - 84:46:fe:89:b9:66 + - 84:46:fe:89:b9:67 + - 84:46:fe:89:b9:68 + - 84:46:fe:89:b9:69 + - ac:b3:b5:1e:d5:eb + - ac:b3:b5:1e:d5:ec + - ac:b3:b5:1e:d5:ed + - ac:b3:b5:1e:d5:ee + - ac:b3:b5:1e:d6:3b + - ac:b3:b5:1e:d6:3c + - ac:b3:b5:1e:d6:3d + - ac:b3:b5:1e:d6:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.2 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

2 4

[PATCH lkp-tests] lib/job.rb: delete fields that need to be reloaded for reproduction task
by Li Ping 01 Feb '21

01 Feb '21

[why] 1. load job from result's job yaml has key "my_email", like my_email: compass-ci-robot(a)qq.com my_name: compass-ci-robot lab: z9 2. load job from result's job yaml has key "SCHED_HOST" which from /etc/compass-ci/defaults/*.yaml, like SCHED_HOST: 172.168.131.2 SCHED_PORT: 3000 each of these fields needs to be reset when submit job.yaml for reprodution task. however, these fields already exist in yaml file, can not to be reload, so delete these fields when submit reproduction job. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 640b25a5f..867c3d193 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -258,6 +258,13 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + if hash.has_key?('id') + hash.delete_if { |key, _| key == "SCHED_HOST" } + hash.delete_if { |key, _| key == "SCHED_PORT" } + hash.delete_if { |key, _| key == "my_name" } + hash.delete_if { |key, _| key == "my_email" } + hash.delete_if { |key, _| key == "lab" } + end revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? -- 2.23.0

3 4

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4

[PATCH compass-ci 4/4] sparrow/5-build: pack lkp-tests
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for user locally deploy compass-ci, submit job will check lkp-$arch.cgz exist, so call container lkp-initrd/run in sparrow script, it is convenient and does not need users to operate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources index 48a740b..cea3b00 100755 --- a/sparrow/5-build/sources +++ b/sparrow/5-build/sources @@ -3,3 +3,5 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. $CCI_SRC/sbin/download-rootfs + +$CCI_SRC/container/lkp-initrd/run -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2

[PATCH lkp-tests 3/3] distro/depends: add monitor-job depends
by Wei Jihui 30 Jan '21

30 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/monitor-job | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/monitor-job diff --git a/distro/depends/monitor-job b/distro/depends/monitor-job new file mode 100644 index 00000000..5bfafa87 --- /dev/null +++ b/distro/depends/monitor-job @@ -0,0 +1,3 @@ +git +ruby-dev +build-essential -- 2.23.0

1 0

[PATCH lkp-tests 2/3] jobs: add monitor-job.yaml
by Wei Jihui 30 Jan '21

30 Jan '21

[usage] add this to CCI_SRC/sbin/auto_submit.yaml, like: repo: - submit monitor-job.yaml os=... monitor_repo=x job_yaml=x job_testbox=x Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/monitor-job.yaml | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 jobs/monitor-job.yaml diff --git a/jobs/monitor-job.yaml b/jobs/monitor-job.yaml new file mode 100644 index 00000000..9b4d4d69 --- /dev/null +++ b/jobs/monitor-job.yaml @@ -0,0 +1,7 @@ +suite: monitor-job +category: functional + +monitor-job: + monitor_repo: + job_yaml: + job_testbox: -- 2.23.0

1 0

[PATCH lkp-tests 1/3] tests: add monitor-job
by Wei Jihui 30 Jan '21

30 Jan '21

monitor-job is for monitor user repos(monitor_repo), if user's repos has new commit, auto submit target job yaml(job_yaml). Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/monitor-job | 63 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100755 tests/monitor-job diff --git a/tests/monitor-job b/tests/monitor-job new file mode 100755 index 00000000..5fd2d73c --- /dev/null +++ b/tests/monitor-job @@ -0,0 +1,63 @@ +#!/bin/sh +# - monitor_repo +# - job_yaml +# - job_testbox + +. $LKP_SRC/lib/log.sh + +set_scheduler_config() +{ + mkdir -p /etc/compass-ci/defaults + cat > /etc/compass-ci/defaults/$lab.yaml <<-EOF + SCHED_HOST: $SCHED_HOST + SCHED_PORT: $SCHED_PORT + lab: $lab + my_email: $my_email + EOF +} + +install_gem_pkgs() +{ + gem sources -c -a https://gems.ruby-china.com/ + gem install activesupport git rest-client elasticsearch +} + +build_my_info() +{ + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git "$CCI_SRC" + ruby "$CCI_SRC"/sbin/build-my-info.rb -e "$my_email" -n "$my_name" -l "$lab" + + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/lkp-tests.git "$LKP_SRC" +} + +download_monitor_repo() +{ + if [ -n "$monitor_repo" ]; then + git clone git://"$LKP_SERVER"/"$monitor_repo" + else + log_error "monitor_repo is not specified" && exit 1 + fi + + test_code=${monitor_repo##*/} + cp -r "$test_code"/* $LKP_SRC/ +} + +submit_job() +{ + yaml=${job_yaml##*/} + "$LKP_SRC"/sbin/submit -a "$yaml" testbox="$job_testbox" +} + +main() +{ + local CCI_SRC=/c/compass-ci + local LKP_SRC=/c/lkp-tests + + install_gem_pkgs + set_scheduler_config + build_my_info + download_monitor_repo + submit_job +} + +main -- 2.23.0

1 0

[PATCH compass-ci] update progress
by Wang Chenglong 30 Jan '21

30 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/support-os-os_arch-os_version.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 6532f92..b354700 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,22 @@ ``` -os os_arch os_version supported-os_mount -openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, - 20.09 cifs/nfs, - mainline-dailybuild cifs/nfs, +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, ,initramfs + 20.09 cifs/nfs, ,initramfs + mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, 20.09 cifs/nfs, -centos aarch64 7 container, - 7.6.1810 cifs/nfs, - 7.8.2003 cifs/nfs, +centos aarch64 7 container + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, +debian aarch64 sid cifs/nfs, ,initfamfs x86 sid cifs/nfs, -archlinux aarch64 2020-11-12 cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot: make sure of consuming delimiter@localhost subqueue at first
by Cao Xueliang 29 Jan '21

29 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index f8d7caa..9e2898e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -75,13 +75,15 @@ class Sched end idle_queues = [] of String + delimiter_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" + delimiter_queues << "#{queue}/delimiter@localhost" end - all_queues = rand_queues(sub_queues) + idle_queues + all_queues = delimiter_queues + rand_queues(sub_queues) + idle_queues - return all_queues + return all_queues.uniq end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: add how to test OSS project
by Wu Fengguang 29 Jan '21

29 Jan '21

Some people apply account by just adding his personal project's git url to upstream-repos. Which is not enough -- we only offer free account to projects that are in our test coverage. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 3e253d0..fda2eba 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -3,7 +3,7 @@ You'll need an account to submit jobs. The account is mainly for tracking your resource consumptions. We have limited pool of testboxes, so there have to be some usage control. -We offer free accounts to OSS (Open Source Software) contributors. +We offer free accounts to OSS (Open Source Software) contributors where the OSS project is in our test coverage. You can prove yourself as OSS contributor by providing "my oss commit" in the below email. We offer free accounts to our collaborators too. If you are one of our @@ -26,10 +26,15 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. + - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. + - my oss commit: a git commit URL that has your name and email. The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). + If your contributed OSS project is not in upstream-repos, you may refer to + [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + to add it to compass-ci testing. ## step 2: receive an email -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/install-tiny: skip steps to deploy compass-ci server
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: execute "./install-tiny", locally deploy compass-ci, can submit job, run containers services, run qemu test. after: execute "./install-tiny server reboot", locally deploy compass-ci server, just run containers servers. [why] some user who locally deploy compass-ci, the host which just run containers can only access intranet, so skip some steps that need to access internet when deploy, can submit job and run qemu test will in other machine, use $LKP_SRC/setup/simply-ci to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index cb47cf5..414c5bc 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -4,10 +4,19 @@ # For your reference only. # It's better to run the below scripts step by step. +export server=$1 +export action=$2 + +server_skip() +{ + local deploy_script=$1 + [ "$server" != "server" ] && ${deploy_script} +} + [[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) cd $CCI_SRC/sparrow || exit -0-package/install +server_skip 0-package/install 0-package/common 1-storage/tiny 1-storage/permission @@ -15,9 +24,9 @@ cd $CCI_SRC/sparrow || exit 2-network/iptables 2-network/nfs 2-network/cifs -3-code/git +server_skip 3-code/git 3-code/dev-env . /etc/profile.d/compass.sh -4-docker/buildall -5-build/ipxe +4-docker/buildall $action +server_skip 5-build/ipxe 7-systemd/systemd-setup -- 2.23.0

3 7

[PATCH v3 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sbin: add script for download os rootfs
by Liu Yinsi 29 Jan '21

29 Jan '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100644 index 0000000..d5ea328 --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +export os=$1 +export os_arch=$2 +export os_version=$3 + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 1

[PATCH v3 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/submit: README file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..9ce7c25 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,30 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit almost all the linux OS. +# Avoid much repetitive testing work to adapt. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Edit executable file: + + submit + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your localhost. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/submit: submit file
by Luan Shengde 29 Jan '21

29 Jan '21

executable file for container: subject it can be run as the real command submit, when you run it, it will calls the command 'submit' in the container add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..20fdf61 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/submit: build file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 29 Jan '21

29 Jan '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..fa5509d --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0

[PATCH compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 29 Jan '21

29 Jan '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add script for download os rootfs
by Liu Yinsi 29 Jan '21

29 Jan '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100644 index 0000000..3977178 --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=/${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir || exit + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

3 4

[PATCH compass-ci 3/4] container/submit: submit file
by Luan Shengde 29 Jan '21

29 Jan '21

executable file for the container it can be run as the real command submit, when you run it, it will calls the command 'submit' in the container add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..67a27d8 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

2 2

[PATCH compass-ci 2/4] container/submit: build file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 29 Jan '21

29 Jan '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0

[PATCH v3 compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 29 Jan '21

29 Jan '21

add output usage add option -h|--help set default option to -h|--help if no options added Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/{build-my-info.rb => build-my-info} | 12 ++++++++++++ 1 file changed, 12 insertions(+) rename sbin/{build-my-info.rb => build-my-info} (78%) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info similarity index 78% rename from sbin/build-my-info.rb rename to sbin/build-my-info index fd75f51..ca6a1de 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 29 Jan '21

29 Jan '21

add output usage add option -h|--help set default option to -h|--help if no options added Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

2 3

[PATCH v2 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 29 Jan '21

29 Jan '21

[why] when we need reboot a vm, we need get it's pid from hostname, then check job_id is the same as required. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_matrixes.rb: treatment of stddev < 1%
by Lu Weitao 29 Jan '21

29 Jan '21

[Why] if stddev < 1% | == 0 We display nothing for stddev column like bellow, it may confuse user. 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 -5.2% 262249.03 redis.get_avg_throughput_rps 61.47 +5.5% 64.83 redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% [How] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 < 1% -5.2% 262249.03 < 1% redis.get_avg_throughput_rps 61.47 < 1% +5.5% 64.83 < 1% redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7760c93..6c88384 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -72,7 +72,7 @@ def get_values(value_list, success) if length > 1 && average != 0 stddev_percent = ( standard_deviation(value_list, average, length) * 100 / average - ).to_i + ) end { average: average, stddev_percent: stddev_percent, min: sorted[0], max: sorted[-1], sorted: sorted } @@ -204,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -615,6 +615,8 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 + return " < 1% " if stddev_percent < 1 + percent_str = get_suitable_number_str( stddev_percent.abs, percent_width - 4, -- 2.23.0

1 0

[PATCH compass-ci 1/2] srv-http/start: add os images data download directory
by Cui Yili 29 Jan '21

29 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

3 4

[PATCH v4 compass-ci 3/3] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 2 +- container/result-cifs/start | 2 +- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 6 +++--- 8 files changed, 10 insertions(+), 15 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 0b9601a..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-cifs/start b/container/os-cifs/start index 09ae3d2..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-nfs/start b/container/os-nfs/start index 9539561..1f4ed42 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { log_error "kernel module nfs is missing (try: sudo modprobe nfs)" diff --git a/container/result-cifs/start b/container/result-cifs/start index de38ca2..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..8d6a85a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -5,7 +5,7 @@ CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" +source ${CCI_SRC}/lib/log.sh build_depends() { @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -41,7 +41,7 @@ do_one() [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container: delete redundant log error info
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: log_error "[ERROR] kernel module ..." after: log_error "kernel module ..." [why] log_error(): echo "$(date +'%F %T') [ERROR] -- $*" >&2 when call log_error(), already has complete log format, so delete redundant "[ERROR]" in parmas. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 4 ++-- container/result-cifs/start | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..0b9601a 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..09ae3d2 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..9539561 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -6,11 +6,11 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { - log_error "[ERROR] kernel module nfsd is missing (try: sudo modprobe nfsd)" + log_error "kernel module nfsd is missing (try: sudo modprobe nfsd)" exit 1 } diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..de38ca2 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib: add log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..626a6e4 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"[INFO] %F %T ${0##*/}: $*" +} + +log_debug() +{ + date +"[DEBUG] %F %T ${0##*/}: $*" +} + +log_warn() +{ + date +"[WARN] %F %T ${0##*/}: $*" >&2 +} + +log_error() +{ + date +"[ERROR] %F %T ${0##*/}: $*" >&2 +} -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 ++-- container/os-cifs/start | 4 ++-- container/os-nfs/start | 4 ++-- container/result-cifs/start | 4 ++-- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 8 files changed, 15 insertions(+), 20 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..7577647 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

2 4

[PATCH compass-ci] sparrow/4-docker: set param value
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: action=reboot after: action=run-only [why] new param value more easy to understand. usage 1: ./buildall -> execute $container/build, install first-run and start script usage 2: ./buildall run-only -> just execute $continer/first-run, start script, no build, install script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- sparrow/7-systemd/cci-network | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..3c0b7b0 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then [ -x build ] && ./build [ -x install ] && ./install fi diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..0438935 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -14,4 +14,4 @@ $CCI_SRC/sparrow/2-network/cifs # --restart=always option is not absolutely reliable # container need to start its dependences firstly. -$CCI_SRC/sparrow/4-docker/buildall reboot +$CCI_SRC/sparrow/4-docker/buildall run-only -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..13cb7c0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + + # example: 2021-01-28 + day = t.strftime('%Y-%m-%d') + # example: 2021-01 + month = t.strftime('%Y-%m') + # example: 2021 + year = t.strftime('%Y') return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/defconfig: load config yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

Load "/etc/compass-ci/register/*.yaml", get register account info from cci defaults config. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index fd13312..f9b0824 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -9,6 +9,7 @@ def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', '/etc/compass-ci/accounts/*.yaml', + '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/auto-submit: bind mount volume
by Liu Yinsi 29 Jan '21

29 Jan '21

mount /etc/compass-ci/register, to get auto-submit account info from config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..79463f6 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] sbin/auto-submit: config account yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, lab, my_token from ES, then config yaml [why] store account info into ES, this part already done by container service register-accounts, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..d8c4b5a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,6 +7,8 @@ require 'bunny' require 'json' require 'yaml' require_relative '../lib/build_my_info_client' +require 'set' +require "#{ENV['CCI_SRC']}/container/defconfig" # receive message and auto submit job class AutoSubmit @@ -86,15 +88,30 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def my_email + names = Set.new %w[auto-submit] + + defaults = relevant_defaults(names) + defaults['auto-submit']['my_email'] +end + +def query_account_info + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 29 Jan '21

29 Jan '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

1 0

[PATCH compass-ci] doc: add support-os-os_arch-os_version.md
by Xu Xijian 29 Jan '21

29 Jan '21

This md document contains the currently supporting os in our server. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/support-os-os_arch-os_version.md diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md new file mode 100644 index 0000000..6532f92 --- /dev/null +++ b/doc/support-os-os_arch-os_version.md @@ -0,0 +1,22 @@ +``` +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, + 20.09 cifs/nfs, + mainline-dailybuild cifs/nfs, + 20.03-dailybuild cifs/nfs, + 20.09-dailybuild cifs/nfs, + 20.03-sp1-dailybuild cifs/nfs, + x86 20.03 cifs/nfs, + 20.09 cifs/nfs, +centos aarch64 7 container, + 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, + 8.1.1911 cifs/nfs, + 8 container, + x86 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, +debian aarch64 sid cifs/nfs, + x86 sid cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, +``` -- 2.23.0

3 4

[PATCH v3 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..2d2f3bf 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/:warning:|\.warning$/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/:error:|\.error$|\.exit_code\./i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..95503ed 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) - t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + t = Time.now.to_s + + # example: 2021-01-28 + day = t[0..9] + # example: 2021-01 + month = t[0..6] + # example: 2021 + year = t[0..3] return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker: pack lkp-$arch.cgz
by Liu Yinsi 29 Jan '21

29 Jan '21

Pack lkp-$arch.cgz by deploy script, it is convenient for users to submit jobs, it doesn't need to be executed by the user. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..014036b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -39,6 +39,7 @@ do_one() fi [ -x first-run ] && ./first-run [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run ) [ -n "$LKP_SRC" ] && log_info "finish build $container." started_containers+="$container_name" -- 2.23.0

2 2