January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc: fix format and spell error
by Liu Yinsi 22 Jan '21

22 Jan '21

1. add 2 space fix no newline 2. golbal -> global, verison -> version Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 48 +++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 794e433..c7450fd 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -3,24 +3,24 @@ 在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 ## 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) - >**说明：** - >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 + >**说明：** + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 - >**说明：** + >**说明：** >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -31,8 +31,8 @@ 2. 配置git账号 ```bash - git config --golbal user.name "XXX" - git config --golbal user.email "xxx(a)aa.com" + git config --global user.name "xxx" + git config --global user.email "xxx(a)xxx.com" ``` 3. 设置文件权限并关闭SELINUX @@ -40,8 +40,8 @@ umask 002 && setenforce 0 ``` - >**说明：** - >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 + >**说明：** + >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 4. 创建工作目录并克隆 compass-ci 项目代码 @@ -71,9 +71,9 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下: - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） @@ -107,7 +107,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: if qemu not exist, output and exit
by Xu Xijian 22 Jan '21

22 Jan '21

If there is no qemu in compass-ci's environment, output something to remind users and then exit. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..70d3ca6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -193,6 +193,10 @@ individual_option() -netdev $netdev ) ;; + *) + echo "qemu not found: $qemu" + exit + ;; esac } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 22 Jan '21

22 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 22 Jan '21

22 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] libvirt based on xml file to run vm, so we need parse xml file nokogiri is a excellent tool of parse xml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH v6 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 46 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..7f64faee --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,46 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +main() +{ + file_check + tbox_check + reset_tbox || { + echo "Failed to reboot the testbox: $host_name" + exit 2 + } +} + +main -- 2.23.0

1 0

[PATCH v5 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 +++++++++ tests/reset-tbox | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..ae3cbe52 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,52 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +result_check() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + file_check + tbox_check + reset_tbox + result_check +} + +main -- 2.23.0

2 2

[PATCH compass-ci] sparrow/1-storage: fix failed to set permission
by Liu Yinsi 22 Jan '21

22 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 ~/compass-ci/sparrow/1-storage# ./permission chown: changing ownership of '/srv/initrd': Read-only file system [why] mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr because mount permission for /srv/initrd is read-only, cause set permission failed. [how] for user install compass-ci at their local server, no need to mount directory, they will download rootfs by wget link, so remove it. mount rootfs can move to selftest own tests deploy-cci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 3 --- 1 file changed, 3 deletions(-) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index eb7270d..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,3 @@ dirs=( ) mkdir -p "${dirs[@]}" - -mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr -mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

2 2

[PATCH compass-ci] add a comment, make it easier to understand
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- providers/qemu/kvm.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..9034ccf 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -89,6 +89,7 @@ set_initrd() set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + # when arch='x86_64', use the follwing file: [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd } -- 2.23.0

2 4

[PATCH compass-ci] doc/manual: optimize submit-job.zh.md
by Wu Zhende 22 Jan '21

22 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- doc/manual/submit-job.zh.md | 34 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 22 deletions(-) diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 540d91f..92faa5c 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -8,32 +8,22 @@ submit 命令的作用是提交测试任务。该命令提供了多个选项帮助用户更灵活的提交任务，您可以在命令行直接输入 summit 命令来查看帮助信息，并根据实际需求灵活使用。 - ### 基本用法 -测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。使用如下命令提交测试任务： - -``` -submit iperf.yaml -``` -如下所示： - -```shell -hi8109@account-vm ~% submit iperf.yaml -submit iperf.yaml, got job_id=z9.173924 -``` +测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。 -本文所示的 yaml 文件均已添加 testbox 字段，若您提交的 yaml 文件不包含该字段将报错： +若您的yaml文件不包含testbox字段，直接提交将会报错： ```shell hi8109@account-vm ~% submit iperf.yaml submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' ``` -您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： +因为testbox是必填字段，您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： -``` -submit iperf.yaml testbox=vm-2p8g +```shell +hi8109@account-vm ~% submit iperf.yaml testbox=vm-2p8g +submit iperf.yaml, got job_id=z9.173924 ``` testbox 字段的值指定需要的测试机，可以使用 `ll` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下所示： @@ -115,7 +105,7 @@ options: 使用-o DIR 命令可以将最终生成的yaml文件保存到指定目录 DIR 下。示例如下所示： ``` - submit -o ~/iperf.yaml + submit -o ~/iperf.yaml testbox=vm-2p8g ``` 运行命令之后会在指定目录生成经过 submit 处理过的 yaml 文件。 @@ -126,7 +116,7 @@ options: 示例命令如下： ``` - submit -a iperf.yaml + submit -a iperf.yaml testbox=vm-2p8g ``` ``` @@ -141,13 +131,13 @@ options: 示例命令如下： ``` - submit -m iperf.yaml + submit -m iperf.yaml testbox=vm-2p8g ``` 控制台显示如下： ```shell - hi8109@account-vm ~% submit -m iperf.yaml + hi8109@account-vm ~% submit -m iperf.yaml testbox=vm-2p8g submit iperf.yaml, got job_id=z9.173923 query=>{"job_id":["z9.173923"]} connect to ws://172.168.131.2:11310/filter @@ -173,14 +163,14 @@ options: 示例命令如下： ``` - submit -m -c borrow-1h.yaml + submit -m -c borrow-1h.yaml testbox=vm-2p8g ``` 当我们提交一个申请设备的任务后，会获取到返回的登陆信息，如 `ssh ip -p port`，添加 -c 参数之后不需要我们手动输入 ssh 登陆命令来进入执行机。控制台显示如下： ```shell - hi8109@account-vm ~% submit -m -c borrow-1h.yaml + hi8109@account-vm ~% submit -m -c borrow-1h.yaml testbox=vm-2p8g submit borrow-1h.yaml, got job_id=z9.173925 query=>{"job_id":["z9.173925"]} connect to ws://172.168.131.2:11310/filter -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci January 2021