- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: [ -n "$initrds" ] || exit after: [ -n "$initrds" ] && cat $initrds > initrd || exit [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 4

[PATCH v3 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

[why] start container os-cifs and os-nfs, need to use sudo command, so add it. container/os-cifs/start have follow line: sudo modprobe cifs container/os-nfs/start have follow line: sudo modprobe nfs Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/README.md: fix format error
by Liu Yinsi 19 Jan '21

19 Jan '21

1. fix format indentation. 2. add Link of debugging task, write PKGBUILD Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 89c43d8..794e433 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -4,24 +4,24 @@ ## 准备工作 - 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -93,7 +93,7 @@ 1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… 3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash @@ -107,9 +107,11 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… - 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) - 以查看iperf任务结果为例操如下： +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 + cat output ``` + + >**说明：** + >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: fix format error
by Liu Yinsi 19 Jan '21

19 Jan '21

1. fix format indentation. 2. add Link of debugging task, write PKGBUILD Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 89c43d8..c199397 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -4,24 +4,24 @@ ## 准备工作 - 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -39,7 +39,7 @@ ```bash umask 002 && setenforce 0 ``` - + >**说明：** >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 @@ -70,7 +70,7 @@ ```bash submit iperf.yaml testbox=vm-2p8g ``` - + 执行上述命令正常情况下会提示信息如下: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory @@ -93,7 +93,7 @@ 1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add…, [编写PKGBUILD](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… 3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash @@ -107,9 +107,12 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… - 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) - 以查看iperf任务结果为例操如下： +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… + 以查看iperf任务结果为例操如下 ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 + cat output ``` + + >**说明：** + >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 1

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

container os-cifs and os-nfs will use sudo command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 19 Jan '21

19 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

When init fork_stat{}, first query the es, get states in es if there is. So historical data can be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 19 Jan '21

19 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/log.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 12 ++++++++---- 2 files changed, 40 insertions(+), 4 deletions(-) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 000000000000..56b08f77b3d0 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +die() +{ + log_error "$@" + exit 1 +} + +log() +{ + log_info "$@" +} + +log_info() +{ + echo "$(date +'%F %T') INFO -- $*" +} + +log_debug() +{ + echo "$(date +'%F %T') DEBUG -- $*" +} + +log_warn() +{ + echo "$(date +'%F %T') WARN -- $*" >&2 +} + +log_error() +{ + echo "$(date +'%F %T') ERROR -- $*" >&2 +} diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..a136c8ff45cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -4,6 +4,8 @@ # - nr_cpu # - memory +source $CCI_SRC/lib/log.sh + : ${nr_cpu:=1} : ${memory:=1G} @@ -52,15 +54,17 @@ parse_ipxe_script() check_option_value() { - [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 } + + [ -s "$kernel" ] || die "Can not find kernel file or kernel file is empty: $kernel." # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || die "Can not find any initrd for current vm." } set_initrd() -- 2.23.0

2 2