- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 22 Jan '21

22 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. For testing ansible git repo, those cannot be tested in a unified way, It mainly tests ansible roles git repo, and these roles repo will be more and more. [why] command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. site.yml like as follow: ``` site.yml --- - hosts: localhost roles: - role: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..38ef0430 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,73 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 1

[PATCH lkp-tests 2/2] pkg/nginx-server: add blacklist.conf
by Wei Jihui 22 Jan '21

22 Jan '21

nginx-server test need forbidden nf_conntrack when os setup Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx-server/PKGBUILD | 3 +++ pkg/nginx-server/blacklist.conf | 10 ++++++++++ 2 files changed, 13 insertions(+) create mode 100644 pkg/nginx-server/blacklist.conf diff --git a/pkg/nginx-server/PKGBUILD b/pkg/nginx-server/PKGBUILD index a8a7091b..95810b1a 100644 --- a/pkg/nginx-server/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -20,4 +20,7 @@ package() { cd "${srcdir}/nginx-${pkgver}.${pkgrel}" make install DESTDIR="${pkgdir}" + + mkdir -p "$pkgdir/rootfs/addon/etc/modprobe.d/" + cp "$LKP_SRC/pkg/${pkgname}/blacklist.conf" "$pkgdir/rootfs/addon/etc/modprobe.d/" } diff --git a/pkg/nginx-server/blacklist.conf b/pkg/nginx-server/blacklist.conf new file mode 100644 index 00000000..e8798cc7 --- /dev/null +++ b/pkg/nginx-server/blacklist.conf @@ -0,0 +1,10 @@ +install nf_conntrack /bin/false +blacklist nf_conntrack +blacklist nf_conntrack_ipv6 +blacklist xt_conntrack +blacklist nf_conntrack_ftp +blacklist xt_state +blacklist iptable_nat +blacklist ipt_REDIRECT +blacklist nf_nat +blacklist nf_conntrack_ipv4 -- 2.23.0

1 0

[PATCH lkp-tests 1/2] pkg: rename nginx to nginx-server
by Wei Jihui 22 Jan '21

22 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/{nginx => nginx-server}/PKGBUILD | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename pkg/{nginx => nginx-server}/PKGBUILD (78%) diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx-server/PKGBUILD similarity index 78% rename from pkg/nginx/PKGBUILD rename to pkg/nginx-server/PKGBUILD index 989676c4..a8a7091b 100644 --- a/pkg/nginx/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -1,4 +1,4 @@ -pkgname=nginx +pkgname=nginx-server pkgver=1.14 pkgrel=2 arch=('i386' 'x86_64') @@ -11,13 +11,13 @@ build() { benchmark_path="/lkp/benchmarks/${pkgname}" mkdir -p "${benchmark_path}" - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/nginx-${pkgver}.${pkgrel}" ./configure --with-http_ssl_module --prefix="${benchmark_path}" make -j4 } package() { - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/nginx-${pkgver}.${pkgrel}" make install DESTDIR="${pkgdir}" } -- 2.23.0

1 0

[PATCH compass-ci] doc: fix format and spell error
by Liu Yinsi 22 Jan '21

22 Jan '21

1. add 2 space fix no newline 2. golbal -> global, verison -> version Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 48 +++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 794e433..c7450fd 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -3,24 +3,24 @@ 在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 ## 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) - >**说明：** - >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 + >**说明：** + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 - >**说明：** + >**说明：** >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -31,8 +31,8 @@ 2. 配置git账号 ```bash - git config --golbal user.name "XXX" - git config --golbal user.email "xxx(a)aa.com" + git config --global user.name "xxx" + git config --global user.email "xxx(a)xxx.com" ``` 3. 设置文件权限并关闭SELINUX @@ -40,8 +40,8 @@ umask 002 && setenforce 0 ``` - >**说明：** - >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 + >**说明：** + >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 4. 创建工作目录并克隆 compass-ci 项目代码 @@ -71,9 +71,9 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下: - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） @@ -107,7 +107,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: if qemu not exist, output and exit
by Xu Xijian 22 Jan '21

22 Jan '21

If there is no qemu in compass-ci's environment, output something to remind users and then exit. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..70d3ca6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -193,6 +193,10 @@ individual_option() -netdev $netdev ) ;; + *) + echo "qemu not found: $qemu" + exit + ;; esac } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 22 Jan '21

22 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 22 Jan '21

22 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] libvirt based on xml file to run vm, so we need parse xml file nokogiri is a excellent tool of parse xml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH v6 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 46 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..7f64faee --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,46 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +main() +{ + file_check + tbox_check + reset_tbox || { + echo "Failed to reboot the testbox: $host_name" + exit 2 + } +} + +main -- 2.23.0

1 0

[PATCH v5 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 +++++++++ tests/reset-tbox | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..ae3cbe52 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,52 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +result_check() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + file_check + tbox_check + reset_tbox + result_check +} + +main -- 2.23.0

2 2