- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/0-package: set umask and selinux
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] some container services need umask 002 to work normally, cci-network.service need disabled or permissive selinux, or will have permission denied questions. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..01c32b2 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -27,6 +27,14 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF +cat >> /etc/bashrc <<EOF +umask 002 +EOF + +source /etc/bashrc + +setenforce 0 + server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

2 1

[PATCH compass-ci 3/4] sparrow/5-build: download rootfs
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] for locally deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03 in deploy script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/sources diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/sources @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index eb9b3cb..a5e314a 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -20,4 +20,5 @@ cd $CCI_SRC/sparrow || exit 4-docker/buildall 5-build/ipxe 5-build/register-accounts +5-build/sources 7-systemd/systemd-setup -- 2.23.0

2 1

[PATCH compass-ci 1/4] sbin: add script for download os rootfs
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] for user deploy compass-ci at their local server, need to download os rootfs to run a qemu kvm or physical machine test. deploy script will call it to download one os rootfs openeuler aarch64 20.03, user can also use this script to download other os rootfs if they want. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100755 index 0000000..5ab427b --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/${os}/${os_arch}/${os_version} + +download_rootfs_file() +{ + mkdir -p /srv/os/${rootfs_dir} + + cd /srv/os/$rootfs_dir || exit + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz + + gzip -dc ${os_version}.cgz |cpio -id + rm -rf ${os_version}.cgz +} + +download_rootfs_file -- 2.23.0

2 1

[PATCH compass-ci 4/4] container: add start-depends for register-accounts container
by Cao Xueliang 27 Jan '21

27 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/register-accounts/start-depends diff --git a/container/register-accounts/start-depends b/container/register-accounts/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/register-accounts/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: add register-accounts container build and Dockerfile
by Cao Xueliang 27 Jan '21

27 Jan '21

This container will register some default accounts which will be used in others service. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/Dockerfile | 15 +++++++++++++++ container/register-accounts/build | 6 ++++++ 2 files changed, 21 insertions(+) create mode 100644 container/register-accounts/Dockerfile create mode 100755 container/register-accounts/build diff --git a/container/register-accounts/Dockerfile b/container/register-accounts/Dockerfile new file mode 100644 index 0000000..933c543 --- /dev/null +++ b/container/register-accounts/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport json yaml elasticsearch io-console diff --git a/container/register-accounts/build b/container/register-accounts/build new file mode 100755 index 0000000..11ac01e --- /dev/null +++ b/container/register-accounts/build @@ -0,0 +1,6 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t register-accounts . + -- 2.23.0

1 0

[PATCH compass-ci 1/4] sparrow: delete register-accounts in install-tiny
by Cao Xueliang 27 Jan '21

27 Jan '21

We will use a container to register accounts. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/install-tiny | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index eb9b3cb..a291549 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,5 +19,4 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe -5-build/register-accounts 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add start depends
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] in Dockerfile and build script, all not depends es, so delete, but in start script, need to query es to get account info, so add start depends es. refer: [PATCH v3 compass-ci] sbin/auto-submit: config account yaml Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 - container/auto-submit/start-depends | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends deleted file mode 100644 index 8357fca..0000000 --- a/container/auto-submit/build-depends +++ /dev/null @@ -1 +0,0 @@ -es diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends index 5232abf..38fc696 100755 --- a/container/auto-submit/start-depends +++ b/container/auto-submit/start-depends @@ -1 +1,2 @@ rabbitmq +es -- 2.23.0

1 0

[PATCH lab-z9] add host-file of a7 and a20
by Zhang Dewan 27 Jan '21

27 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a20 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a7 | 25 +++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a20 create mode 100644 hosts/taishan200-2280-2s64p-256g--a7 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 new file mode 100644 index 0000000..21bf8e1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b888f2dd + - /dev/disk/by-id/scsi-350000399b888f705 + - /dev/disk/by-id/scsi-350000399b88906f5 + - /dev/disk/by-id/scsi-350000399b8890141 +mac_addr: + - 84:46:fe:3d:a7:d5 + - 84:46:fe:3d:a7:d6 + - 84:46:fe:3d:a7:d7 + - 84:46:fe:3d:a7:d8 + - 84:46:fe:3d:a7:d9 + - 84:46:fe:3d:a7:da + - 84:46:fe:3d:a7:db + - 84:46:fe:3d:a7:dc +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 new file mode 100644 index 0000000..07701d3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039998134611 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 +mac_addr: + - 84:46:fe:89:be:20 + - 84:46:fe:89:be:21 + - 84:46:fe:89:be:22 + - 84:46:fe:89:be:23 + - ac:b3:b5:1e:d6:eb + - ac:b3:b5:1e:d6:ec + - ac:b3:b5:1e:d6:ed + - ac:b3:b5:1e:d6:ee + - ac:b3:b5:1e:d7:7b + - ac:b3:b5:1e:d7:7c + - ac:b3:b5:1e:d7:7d + - ac:b3:b5:1e:d7:7e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.7 -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: update my_info before store into ES
by Liu Yinsi 27 Jan '21

27 Jan '21

before: when store account info to ES, generate new my_token. after: when store account info to ES, if my_email exist, use old my_token, if not exist, generate new my_token. [why] prevent my_token in config yaml is different with ES. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..6aa842c --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -66,12 +66,16 @@ class BuildMyInfo def store_account_info es = ESClient.new(index: 'accounts') + + my_info_es = es.query_by_id(@my_info['my_email']) || {} + @my_info.update my_info_es + es.put_source_by_id(@my_info['my_email'], @my_info) end def config_my_info + store_account_info config_default_yaml config_lab_yaml - store_account_info end end -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..91c659d 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,26 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 1

[PATCH compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..2532988 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,25 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 2

[PATCH lkp-tests] stats: add output failed msg
by Zhang Yale 27 Jan '21

27 Jan '21

[Why] Output meed to be displayed in the ansible_test.json file. [Input] fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define mysql_packages.\n ^ here\n"} PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 [Output] playbook.run.msg: The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: Define mysql_packages. ^ here nr_ok: 2 nr_changed: 0 nr_failed: 1 nr_skipped: 1 nr_rescued: 0 nr_ignored: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3..f0766412 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,17 @@ #!/usr/bin/env ruby +require 'json' while (line = STDIN.gets) case line.chomp! + when /FAILED!\s+=>\s+(\{.*\})/ + value = JSON.load($1)['msg'] + puts "playbook.run.msg: #{value}" when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + puts "nr_ok: #{$1}" + puts "nr_changed: #{$2}" + puts "nr_failed: #{$4}" + puts "nr_skipped: #{$5}" + puts "nr_rescued: #{$6}" + puts "nr_ignored: #{$7}" end end -- 2.23.0

2 2

[PATCH compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 27 Jan '21

27 Jan '21

add output usage add option -h|--help set default option to -h|--help Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] providers/libvirt: split common elements of domain.xml
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] some elements can be specifies by user. case 1: a complete domain.xml we should replace some elements of user's xml like os name ... case 2: default domain.xml generate a default domain.xml case 3: only specifies a cpu templates use user's cpu template replace default cpu template case 4: composition template user can specifies different tempaltes like cpu, disk, ... [how] split domain.xml then template combination based on user requirements Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 6 ++++++ providers/libvirt/templates/cpu.xml | 6 ++++++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 13 files changed, 60 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..7646c91 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,6 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> + diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..6991457 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,6 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +  +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..317866d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'><%= loader %></loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-fail-email
by Luan Shengde 27 Jan '21

27 Jan '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..bcc4326 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Get a commit than was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + We can not find such commit ID from the repo logs - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit id correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: fix bug for parsing email
by Luan Shengde 27 Jan '21

27 Jan '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. so users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..95b967a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,34 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(git -C #{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C #{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf repo_dir end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'] - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: replace gem sources
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] function linux_dep() and mac_dep() all use 'gem install', so replace gem sources in function entry run(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..32d71ccf8 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -39,6 +39,7 @@ mac_dep() run() { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ if is_system "Linux"; then linux_dep elif is_system "Darwin"; then -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: modify the incorrect link
by Li Ping 27 Jan '21

27 Jan '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sparrow/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 1906f10..703e8f9 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -78,11 +78,11 @@ ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 -4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） +4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.160.11.58:11300/os/)下获取对应版本的cgz文件） ```bash mkdir -p /srv/os/openeuler/aarch64/20.03 cd /srv/os/openeuler/aarch64/20.03 - wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz + wget http://api.compass-ci.openeuler.org:11300/os/openeuler/aarch64/20.03.cgz ``` 5. 解压rootfs cgz 文件 @@ -109,7 +109,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

2 3

[PATCH v2 compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 27 Jan '21

27 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **登录测试环境** -1. 向 compass-ci-reply(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci-robot(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… -- 2.23.0

1 0

[PATCH lkp-tests] pkg/mysql/PKGBUILD: fix mysql pkgbuild
by Wei Jihui 27 Jan '21

27 Jan '21

[error] CMake Error at plugin/group_replication/libmysqlgcs/rpcgen.cmake:100(MESSAGE): Could not find rpcgen ... -- Configuring incomplete, errors occurred! [how] make and make install rpcgen Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 8869e62b..0f7d8bc8 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -7,8 +7,9 @@ license=('GPLv2') source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "0000-mysql-add-fstack-protector-strong.patch" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') prepare() { @@ -19,9 +20,18 @@ prepare() build() { build_getcpu + build_rpcsvc build_mysql } +build_rpcsvc() +{ + cd "$srcdir/rpcsvc-proto-1.4" + ./configure + make + make install +} + build_mysql() { cd $srcdir/$pkgname-$pkgver -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix.rb: fix NoMethodError
by Lu Weitao 27 Jan '21

27 Jan '21

[Why] /home/wangyong/compass-ci/lib/matrix2.rb:21:in `set_pre_value': undefined method `sum' for 1:Integer (NoMethodError) Did you mean? succ root cause: if a job has *.yaml(except job.yaml) in result_root, will meet this error when merge it to stats.json directly. [How] remove the logical handle *.yaml from lib/matrix.rb at first, we should only handle $script.yaml, and the feature will be done in other patch Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..4dd03b4 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,21 +39,19 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') - when /\.yaml$/ - File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if %w[stats job].include?(monitor) + next if monitor == 'stats' extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 27 Jan '21

27 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: [before] EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: [after] ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

1 0

[PATCH compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs.rb: fix error if jobs do not contain stats
by Wu Zhende 27 Jan '21

27 Jan '21

[Before] Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] kvcount.category=benchmark: 1 kvcount.os=openeuler: 1 kvcount.suite=iperf: 1 kvcount.tbox_group=vm-2p16g: 1 raw.id.[arch=aarch64]: ["crystal.786605"] raw.id.[category=benchmark]: ["crystal.786605"] raw.id.[os=openeuler]: ["crystal.786605"] raw.id.[suite=iperf]: ["crystal.786605"] raw.id.[tbox_group=vm-2p16g]: ["crystal.786605"] Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index eb3f70d..e367a91 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -84,7 +84,7 @@ class ESJobs metrics.concat(stats.keys) end - metrics.uniq! + metrics.uniq end def initialize_result_hash(metrics) @@ -137,11 +137,11 @@ class ESJobs metrics = get_all_metrics(jobs) result = initialize_result_hash(metrics) jobs.each do |job| + kvcount(result, job) stats = job['stats'] next unless stats set_default_value(result, stats, metrics) - kvcount(result, job) end stats_count(result) -- 2.23.0

2 1

[PATCH v5 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 27 Jan '21

27 Jan '21

Add limit for borrowing machine: borrowing period <= 10 days Only jobs that runtime/sleep value is less than or equal to 10 days are allowed to borrow machines. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..5448082 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,19 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime value is more than 10 days, job will fail. + raise error_msg if hash["runtime"].to_i > max_run_time + raise error_msg if hash["sleep"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: fix install sysbench pkg depends
by Wei Jihui 27 Jan '21

27 Jan '21

[error] Error: Unable to find a match: mysql-devel [why] openeuler 20.03 has not libmysqlclient-dev: mysql-devel change it to libmariadbclient-dev Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-dev | 2 +- distro/depends/sysbench-mysql-dev | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/distro/depends/sysbench-dev b/distro/depends/sysbench-dev index 3fa67deb..f570c463 100644 --- a/distro/depends/sysbench-dev +++ b/distro/depends/sysbench-dev @@ -1,2 +1,2 @@ pkg-config -libmysqlclient-dev +libmariadbclient-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index ca5b03d8..65633b33 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,3 +1,2 @@ pkg-config -libmysqlclient-dev mysql-server -- 2.23.0

2 1

[PATCH compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..1ea36b0 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

2 4

[PATCH lkp-tests] jobs: add multi-qemu.yaml multi-docker.yaml
by Xiao Shenwei 26 Jan '21

26 Jan '21

submit the job to a physical machine, then the machine will start some clients to consume job. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- jobs/multi-docker.yaml | 17 +++++++++++++++++ jobs/multi-qemu.yaml | 23 +++++++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 jobs/multi-docker.yaml create mode 100644 jobs/multi-qemu.yaml diff --git a/jobs/multi-docker.yaml b/jobs/multi-docker.yaml new file mode 100644 index 000000000..8e42208a9 --- /dev/null +++ b/jobs/multi-docker.yaml @@ -0,0 +1,17 @@ +suite: multi-docker +category: functional +simplify-ci: +runtime: 100d +multi-docker-0: + nr_container: 20 + tbox_group: dc-1g +multi-docker-1: + nr_container: 10 + tbox_group: dc-2g +multi-docker-2: + nr_container: 5 + tbox_group: dc-4g +multi-docker-3: + nr_container: 20 + tbox_group: dc-8g +sleep: diff --git a/jobs/multi-qemu.yaml b/jobs/multi-qemu.yaml new file mode 100644 index 000000000..200f66065 --- /dev/null +++ b/jobs/multi-qemu.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g +sleep: -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 60 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..b4f8b4c --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,60 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def job_exist?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request_job(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if job_exist?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request_job(context, sched_client, logger) + return if response.nil? + + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

1 0

[PATCH compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 67 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..0ff0d4f --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,67 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def nojob?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if nojob?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request(context, sched_client, logger) + return if response.nil? + + # TODO + # based on response load file + # TODO + # generate domain.xml + # TODO + # use ruby-libvirt api start a vm + # clean env + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

3 4

[PATCH lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 26 Jan '21

26 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

2 1

[PATCH compass-ci] doc/manual: optimize borrow-machine.en.md
by Luan Shengde 26 Jan '21

26 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.en.md | 341 +++++++++----------------------- 1 file changed, 98 insertions(+), 243 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 6a1fe19..36823c3 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -1,246 +1,101 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - -- Send an email to apply for an account. -- Receive an email from compass-ci-robot(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +# Prepare + +- Apply account +- Config default yaml files + +If have not completed above works, reference to [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to finish it. + +# Apply testbox + +## 1. Generate local ssh pub key + + Use the command below to check the pub key exists: + + ls ~/.ssh/*.pub + + Generate one if you don't have one now: + + ssh-keygen + +## 2. Select the job yaml + + We have provided various examples for you in dir ~/lkp-tests/jobs: + + You can filtrate examples for borrowing machine with command below: + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. Submit job + + Command to submit jobs: + + for DCs: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + for VMs: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + for HWs: + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - You can view the logs in real time for the job with the command above。 + - You will directly login the testbox if the job runs successfully. + - And you will receive an email that contains login command and server configuration information. + - Only within the period of borrowing machine, you can access the textbox with the login command. + +## 4. Return testbox + + Reruan manually(recommended): + + Manually execute 'reboot' in time to return the testbox. + Avoid a waste of computer resource with no-load running. + + Automatically return on maturity: + + The testbox will be returned automatically if it has expired its service life. + + - All testboxes will be returned if it has been executed command 'reboot'. + - The testbox cannot be accessed any more after it has been returned. + - Apply a new one if you want continue to use it. # FAQ -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the runtime field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit-job.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. +## Customize the borrowing time + + Find key in the yaml file and edit its value according to your requirement. + + - The borrowing period can be calculated in days and hours. + - The maximum period is no more than 10 day. + +## Guidance for command submit + + See the usage and options for command 'submit' the command below: + + submit -h + + Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… + +## Available testbox + + For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **instruction:** + > + > - DCs: dc-xxx + > - VMs: vm-xxx + > - HWs: taishan200-2280-xxx + + >![](./../public_sys-resources/icon-notice.gif) **attention:** + > - It means that you choosed a specified physical machine if the testbox name is end with `--axx`. + > - You will need to wait if there are already tasks in the task queue for the machine. + > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. + +## Specify the OS + + About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… -- 2.23.0

1 0

[PATCH lkp-tests] lib/log.sh: add square bracket
by Liu Yinsi 26 Jan '21

26 Jan '21

before: 2021-01-26 14:53:36 ERROR -- cifs module is not loaded in after: 2021-01-26 15:00:27 [ERROR] -- cifs module is not loaded in [why] use square bracket make log more obvious. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/log.sh b/lib/log.sh index 5acf61825..207795ece 100644 --- a/lib/log.sh +++ b/lib/log.sh @@ -2,25 +2,25 @@ log() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_info() { - echo "$(date +'%F %T') INFO -- $*" + echo "$(date +'%F %T') [INFO] -- $*" } log_debug() { - echo "$(date +'%F %T') DEBUG -- $*" + echo "$(date +'%F %T') [DEBUG] -- $*" } log_warn() { - echo "$(date +'%F %T') WARN -- $*" >&2 + echo "$(date +'%F %T') [WARN] -- $*" >&2 } log_error() { - echo "$(date +'%F %T') ERROR -- $*" >&2 + echo "$(date +'%F %T') [ERROR] -- $*" >&2 } -- 2.23.0

1 0

[PATCH lkp-tests] feat: support a new field for local disk: rootfs_disk
by Yu Chuan 26 Jan '21

26 Jan '21

[Why] We now need auto install iso to the specific disk of testbox, so in order to avoid conflict with rootfs_partition, we support a new field for local disk: rootfs_disk. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- filters/run_on_local_disk | 8 ++++---- lib/bootstrap.sh | 23 +++++++++++++++++++++++ lkp-exec/qemu | 7 ++++++- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/filters/run_on_local_disk b/filters/run_on_local_disk index 389d1ca17fdc..2092bd2b6028 100755 --- a/filters/run_on_local_disk +++ b/filters/run_on_local_disk @@ -1,11 +1,11 @@ #!/bin/bash -# check if testbox has rootfs_partition to run the job +# check if testbox has rootfs_partition or rootfs_disk to run the job if [[ "$run_on_local_disk" == "yes" || "$run_on_local_disk" == "true" ]]; then - if [[ -z "$rootfs_partition" ]]; then - echo "rm $job_file due to no rootfs_partition" - echo "run_on_local_disk needs rootfs_partition" + if [[ -z "$rootfs_partition" && -z "$rootfs_disk" ]]; then + echo "rm $job_file due to no rootfs_partition or rootfs_disk" + echo "run_on_local_disk needs rootfs_partition or rootfs_disk" exit 1 fi fi diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd48b7..6c0055bd19f5 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -490,6 +490,12 @@ wait_load_disk() ls $rootfs_partition >/dev/null 2>&1 && return # eg: LABEL=LKP-ROOTFS blkid | grep -q ${rootfs_partition#*=} && return + + # eg: /dev/disk/by-id/ata-WDC_WD1002FAEX-00Z3A0_WD-WCATRC577623 + ls $rootfs_disk >/dev/null 2>&1 && return + # eg: LABEL=LKP-ROOTFS + blkid | grep -q ${rootfs_disk#*=} && return + sleep 1 done @@ -515,6 +521,23 @@ mount_rootfs() mkdir -p $ROOTFS_DIR/tmp CACHE_DIR=$ROOTFS_DIR/tmp cleanup_pkg_cache $CACHE_DIR + elif [ -n "$rootfs_disk" ]; then + # wait for the machine to load the disk + wait_load_disk || { + # skipping following test if disk can't be load + echo "can't load the disk $rootfs_disk, skip testing..." + set_job_state 'load_disk_fail' + return 1 + } + ROOTFS_DIR=/opt/rootfs + mkdir -p $ROOTFS_DIR + mount $rootfs_disk $ROOTFS_DIR || { + mkfs.btrfs -f $rootfs_disk + mount $rootfs_disk $ROOTFS_DIR + } + mkdir -p $ROOTFS_DIR/tmp + CACHE_DIR=$ROOTFS_DIR/tmp + cleanup_pkg_cache $CACHE_DIR else CACHE_DIR=/tmp/cache mkdir -p $CACHE_DIR diff --git a/lkp-exec/qemu b/lkp-exec/qemu index 9e4b576cf50f..1cbd316f6146 100755 --- a/lkp-exec/qemu +++ b/lkp-exec/qemu @@ -91,14 +91,16 @@ replace_script_partition_val() local nr_hdd_vdisk="$(echo $hdd_partitions | wc -w)" local nr_ssd_vdisk="$(echo $ssd_partitions | wc -w)" local nr_swap_vdisk="$(echo $swap_partitions | wc -w)" + local nr_rootfs_vdisk="$(echo $rootfs_partition | wc -w)" + [ -z "$rootfs_partition" ] && nr_rootfs_vdisk="$(echo $rootfs_disk | wc -w)" [[ $nr_hdd_partitions ]] || nr_hdd_partitions=$nr_hdd_vdisk [[ $nr_ssd_partitions ]] || nr_ssd_partitions=$nr_ssd_vdisk VDISK_NUM=$((nr_hdd_partitions+nr_ssd_partitions+nr_swap_vdisk+nr_rootfs_vdisk)) - [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition" =~ '/dev/vda' ]] && return + [[ "$hdd_partitions$ssd_partitions$swap_partitions$rootfs_partition$rootfs_disk" =~ '/dev/vda' ]] && return if ((VDISK_NUM)); then local index=0 @@ -115,6 +117,9 @@ replace_script_partition_val() -e "s%export swap_partitions=.*%export swap_partitions='${vdisk_swap_val}'%" \ -e "s%export rootfs_partition=.*%export rootfs_partition='${vdisk_rootfs_val}'%" \ $job_script + + [ -z "$rootfs_partition" ] && + sed -i -e "s%export rootfs_disk=.*%export rootfs_disk='${vdisk_rootfs_val}'%" $job_script fi } -- 2.23.0

3 5

[PATCH compass-ci] update apply account email address
by Li Yuanchao 26 Jan '21

26 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH v13 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 26 Jan '21

26 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 40 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.last_state.test.iperf.exit_code.127: 12 sum.stats.stderr./lkp/lkp/src/lib/env.sh:#:set_perf_path: 12 sum.stats.last_state.is_incomplete_run: 12 sum.stats.kmsg.timestamp:last: 45 sum.stats.last_state.exit_fail: 20 sum.stats.stderr.has_stderr: 44 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 213 ++++++++++++++++++++++++++++--------------------- 1 file changed, 123 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c35b182 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,170 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) - - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value + def set_job_summary(stats, job) + summary_result = '' + stats.each_key do |stat| + # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, + # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, + # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, + if stat.match(/warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "stderr.linux-perf": 1, + # "stderr.error:target_not_found:ruby-dev": 1, + # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, + if stat.match(/stderr\./i) + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, + # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, + # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + # "last_state.test.iperf.exit_code.127": 1, + # "last_state.test.cci-makepkg.exit_code.1": 1, + if stat.match(/error|\.exit_code\./i) + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stat.match(/\.fail$/i) + job['summary.any_fail'] = 1 + summary_result = 'fail' end - - next unless job['stats'] - - @result[field] << job['stats'][field] if job['stats'][field] end - end - - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + return unless summary_result.empty? - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) + job['summary.success'] = 1 + end - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + set_job_summary(stats, job) end - - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + metrics.concat(stats.keys) end - return level + metrics.uniq! end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end - key = job_id + '.' + key - @stats_filter_result[key] = value + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - break + stats.each do |key, value| + result['raw.stats'][key] << value end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + end + end - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 2

[PATCH compass-ci] lib/git: use work_dir replace repo for GitCommit class
by Cao Xueliang 26 Jan '21

26 Jan '21

If we use "git -C /srv/git/#{repo}.git", we need mount "/srv/git", but sometimes we have clone the git repo, so we don't need mount "/srv/git". Now, we just need to pass the work_dir to the GitCommit class. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git.rb | 4 ++-- lib/git_bisect.rb | 2 +- lib/mail_bisect_result.rb | 8 +++++++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index db97c47..9751dcf 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,8 +2,8 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) - @git_prefix = "git -C /srv/git/#{repo}.git" + def initialize(work_dir, commit) + @git_prefix = "git -C #{work_dir}" @commit = commit end diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index e577afd..5d13074 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -100,10 +100,10 @@ class GitBisect bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, + 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, 'pkgbuild_repo' => @pkgbuild_repo, diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 209a28d..637c008 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -12,12 +12,13 @@ require_relative 'assign_account_client' class MailBisectResult def initialize(bisect_info) @repo = bisect_info['repo'] + @work_dir = bisect_info['work_dir'] @commit_id = bisect_info['commit'] @all_errors = bisect_info['all_errors'] @bisect_error = bisect_info['bisect_error'] @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] - @git_commit = GitCommit.new(@repo, @commit_id) + @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' @@ -26,6 +27,7 @@ class MailBisectResult def create_send_email send_report_mail(compose_mail) send_account_mail + rm_work_dir end def compose_mail @@ -80,4 +82,8 @@ class MailBisectResult apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end + + def rm_work_dir + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) + end end -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: fix error when search nothing
by Wu Zhende 26 Jan '21

26 Jan '21

[Before] es-jobs suite=test Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] es-jobs suite=test No query result is found: {"suite"=>"test"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index c35b182..eb3f70d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -13,7 +13,8 @@ require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @jobs = query_jobs_from_es(es_query) + @es_query = es_query + @jobs = query_jobs_from_es @refine = my_refine @fields = fields @stats_filter = stats_filter @@ -21,9 +22,9 @@ class ESJobs set_jobs_summary end - def query_jobs_from_es(items) + def query_jobs_from_es es = ESQuery.new(ES_HOST, ES_PORT) - result = es.multi_field_query items + result = es.multi_field_query @es_query jobs = result['hits']['hits'] jobs.map! { |job| job['_source'] } return jobs @@ -164,6 +165,10 @@ class ESJobs end def output + if @jobs.empty? + puts "No query result is found: #{@es_query}" + return + end @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: the repo should be in our test coverage
by Wu Fengguang 26 Jan '21

26 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index b5430e7..3e253d0 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -27,8 +27,9 @@ Email template: - please replace the 3 parts in {{ }} with your information. - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your contribution. It should be in a - well known git repo that's registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) +- my oss commit: a git commit URL that has your name and email. + The git repo should be in our test coverage and registered in + [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). ## step 2: receive an email -- 2.23.0

1 0

[PATCH v6 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler [how] each method corresponds to a interface, so ensuring the independence of them. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..93f9ccd 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,37 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + def register_mac2host(hostname, mac) + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{hostname}&mac=#{mac}", {} + ) + end + + def register_host2queues(hostname, queues) + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{hostname}&queues=#{queues}", {} + ) + end + + def delete_mac2host(mac) + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{mac}", {} + ) + end + + def delete_host2queues(hostname) + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{hostname}", {} + ) + end end -- 2.23.0

1 0

[PATCH lab-z9] add hosts-file of taishan200-2280-2s64p-256g--a5 which include NMVe SSD
by Zhang Dewan 26 Jan '21

26 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a5 | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a5 diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 new file mode 100644 index 0000000..fc9a594 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -0,0 +1,26 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981306c5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43957 + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 +mac_addr: + - 84:46:fe:26:d3:47 + - 84:46:fe:26:d3:48 + - 84:46:fe:26:d3:49 + - 84:46:fe:26:d3:4a + - ac:b3:b5:1e:b4:ab + - ac:b3:b5:1e:b4:ac + - ac:b3:b5:1e:b4:ad + - ac:b3:b5:1e:b4:ae + - ac:b3:b5:1e:ba:9b + - ac:b3:b5:1e:ba:9c + - ac:b3:b5:1e:ba:9d + - ac:b3:b5:1e:ba:9e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.5 -- 2.23.0

2 2

[PATCH v5 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..a81ffbb 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,41 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH v2 compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 3 ++- container/os-cifs/start | 3 ++- container/os-nfs/start | 6 ++++-- container/result-cifs/start | 3 ++- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..c70b318 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..6df167f 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..3b0ebb5 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -5,10 +5,12 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + echo "[ERROR] nfs module is not loaded in the Docker host's kernel (try: sudo modprobe nfs)" + exit } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + echo "[ERROR] nfsd module is not loaded in the Docker host's kernel (try: sudo modprobe nfsd)" + exit } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..3992cdd 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm result-cifs -- 2.23.0

2 4

[PATCH compass-ci] sparrow/3-code: set params
by Liu Yinsi 26 Jan '21

26 Jan '21

before: deploy compass-ci client and server in one machine. after: not only deploy compass-ci client and server in one machine, but also on different servers. [why] for user locally deploy compass-ci, sometimes need to deploy client in other machine, so add $1. it's interrelated with the follow patch: [PATCH lkp-tests] setup/simplify-ci: delete default value Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 2ba9662..00a2e0f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -4,6 +4,9 @@ server_name=$(hostname | cut -f1 -d.) server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + +[ -n "$1" ] && server_ip=$1 + : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${os_http_host:=$server_ip} -- 2.23.0

1 0

[PATCH v2 lkp-tests] lkp-bootstrap.service: fix killmode error
by Wang Chenglong 26 Jan '21

26 Jan '21

[Error] Welcome to openEuler 20.09! /usr/lib/systemd/system/lkp-bootstrap.service:11: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the ser vice. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. [How] config the lkp-bootstrap.service: KillMode=control-group Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service index 842c4aef8..a9eb5152a 100644 --- a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service +++ b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service @@ -8,7 +8,7 @@ Type=forking Environment=HOME=/root ExecStart=/etc/init.d/lkp-bootstrap IgnoreSIGPIPE=no -KillMode=none +KillMode=control-group TasksMax=infinity [Install] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: add introduction
by Wu Fengguang 26 Jan '21

26 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index a0b2c82..b5430e7 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -1,5 +1,15 @@ # apply account +You'll need an account to submit jobs. The account is mainly for tracking your resource consumptions. +We have limited pool of testboxes, so there have to be some usage control. + +We offer free accounts to OSS (Open Source Software) contributors. +You can prove yourself as OSS contributor by providing "my oss commit" in the below email. + +We offer free accounts to our collaborators too. If you are one of our +collaborators, you should know the collaboration channel to apply account. +In this case you just need provide name, email, ssh pubkey and purpose. + ## step 1: send email to apply account Email template: @@ -16,8 +26,9 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. -- the email name should be in english, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your contribution. +- the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. +- my oss commit: a git commit URL that has your contribution. It should be in a + well known git repo that's registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos) ## step 2: receive an email @@ -29,7 +40,7 @@ Which contains account information for you: ## step 3: one-time setup -**Follow the email** to setup your local environment: +**Follow instructions in the email** to setup your local environment: git clone https://gitee.com/wu_fengguang/lkp-tests.git setup ~/.config/compass-ci/defaults/account.yaml @@ -38,10 +49,3 @@ Which contains account information for you: Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -## notice: - -In case that you received a failed email with message as below: - - The repo url for your commit is not in our upstream-repo list. - -Try to [register repo to upstream-repos](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… -- 2.23.0

1 0

[PATCH compass-ci] uniform coding style
by Li Ping 26 Jan '21

26 Jan '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 12c1f3e..7760c93 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -204,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

1 0

[PATCH v3 compass-ci 5/5] upload-libvirt-xml/nginx.conf: nginx.conf in upload-libvirt-xml container
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/nginx.conf | 43 +++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt-xml/nginx.conf diff --git a/container/upload-libvirt-xml/nginx.conf b/container/upload-libvirt-xml/nginx.conf new file mode 100644 index 0000000..aef8fb8 --- /dev/null +++ b/container/upload-libvirt-xml/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt-xml"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

3 2

[PATCH v2 compass-ci] container: fix maintainer email address format
by Liu Yinsi 26 Jan '21

26 Jan '21

1. maintainer email address format use: xxx <xxx(a)xxx.com> 2. Update maintainers, shellcheck for Bai Jing <799286817(a)qq.com> web-backend-nginx and web-backend for Lu Weitao <luweitaobe(a)163.com> FYI, Baijing and Weitao, please help to maintain the containers. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assistant/Dockerfile | 2 +- container/send-internet-mail/Dockerfile | 2 +- container/send-mail/Dockerfile | 2 +- container/shellcheck/Dockerfile | 2 +- container/web-backend-nginx/Dockerfile | 2 +- container/web-backend/Dockerfile | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..ce124b9 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -3,7 +3,7 @@ FROM alpine:3.11 -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile index f1ad7a2..04d2da0 100644 --- a/container/assistant/Dockerfile +++ b/container/assistant/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index 03e3327..29f1bc6 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index d151bda..b334b07 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/shellcheck/Dockerfile b/container/shellcheck/Dockerfile index b78beb7..0bfe7c0 100644 --- a/container/shellcheck/Dockerfile +++ b/container/shellcheck/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER 18509160991(a)163.com +MAINTAINER Bai Jing <799286817(a)qq.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/web-backend-nginx/Dockerfile b/container/web-backend-nginx/Dockerfile index a3882ac..bbf7af2 100644 --- a/container/web-backend-nginx/Dockerfile +++ b/container/web-backend-nginx/Dockerfile @@ -3,6 +3,6 @@ FROM nginx:alpine -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> COPY nginx.conf /etc/nginx/ diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 2ac5148..a26c167 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -2,7 +2,7 @@ FROM debian -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> ENV DEBIAN_FRONTEND noninteractive -- 2.23.0

1 0

[PATCH compass-ci] container: Fix maintainer email address format
by Liu Yinsi 26 Jan '21

26 Jan '21

1. maintainer email address format use: xxx <xxx(a)xxx.com> 2. Update maintainers, shellcheck for Bai Jing <799286817(a)qq.com> web-backend-nginx and web-backend for Lu Weitao <luweitaobe(a)163.com> FYI, Baijing and Weitao, please help to maintain the containers. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assistant/Dockerfile | 2 +- container/send-internet-mail/Dockerfile | 2 +- container/send-mail/Dockerfile | 2 +- container/shellcheck/Dockerfile | 2 +- container/web-backend-nginx/Dockerfile | 2 +- container/web-backend/Dockerfile | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..ce124b9 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -3,7 +3,7 @@ FROM alpine:3.11 -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile index f1ad7a2..04d2da0 100644 --- a/container/assistant/Dockerfile +++ b/container/assistant/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER caoxl78320(a)163.com +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index 03e3327..29f1bc6 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index d151bda..b334b07 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER shdluan(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ diff --git a/container/shellcheck/Dockerfile b/container/shellcheck/Dockerfile index b78beb7..0bfe7c0 100644 --- a/container/shellcheck/Dockerfile +++ b/container/shellcheck/Dockerfile @@ -3,7 +3,7 @@ FROM debian -MAINTAINER 18509160991(a)163.com +MAINTAINER Bai Jing <799286817(a)qq.com> ENV DEBIAN_FRONTEND noninteractive diff --git a/container/web-backend-nginx/Dockerfile b/container/web-backend-nginx/Dockerfile index a3882ac..bbf7af2 100644 --- a/container/web-backend-nginx/Dockerfile +++ b/container/web-backend-nginx/Dockerfile @@ -3,6 +3,6 @@ FROM nginx:alpine -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> COPY nginx.conf /etc/nginx/ diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 2ac5148..a26c167 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -2,7 +2,7 @@ FROM debian -MAINTAINER Zhang Yuhang <o0OiIl(a)hotmail.com> +MAINTAINER Lu Weitao <luweitaobe(a)163.com> ENV DEBIAN_FRONTEND noninteractive -- 2.23.0

1 1

[PATCH v3 compass-ci 1/5] upload-libvirt-xml/build: add upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/build | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 container/upload-libvirt-xml/build diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build new file mode 100755 index 0000000..d39838b --- /dev/null +++ b/container/upload-libvirt-xml/build @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# The libvirt xml upload has nothing to do with the job. It renames the uploaded file and does not currently have permission to upload. +# it is a transitional service. Two different functions use the same port service, + +docker build -t upload-libvirt-xml . -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..db60fa2 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type: '', parameter: '', value: '') + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

3 4

[PATCH v4 compass-ci 2/2] container/lifecycle: add a new docker service
by Wu Zhende 26 Jan '21

26 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 +++++++++++++++++++++++ container/lifecycle/build | 18 +++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 33 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 78 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..aa3e737 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport git json yaml + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..ba68a85 --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..7e67b3c --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] service/lifecycle: add a new service
by Wu Zhende 26 Jan '21

26 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..af5cd07 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "Lifecycle Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/sched_client.rb: add some common methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..b317790 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type, parameter, value) + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH v4 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Jan '21

26 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs that runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..2c3007c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,18 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime value is more than 10 days, job will fail. + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH v3 compass-ci 3/5] upload-libvirt-xml/start: start upload-libvirt-xml docker container
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/upload-libvirt-xml/start diff --git a/container/upload-libvirt-xml/start b/container/upload-libvirt-xml/start new file mode 100755 index 0000000..ddb04e7 --- /dev/null +++ b/container/upload-libvirt-xml/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +: ${UPLOAD_LIBVIRT_PORT:=3081} + +docker_rm upload-libvirt-xml + +cmd=( + docker run -d + --name upload-libvirt-xml + --restart=always + -p ${UPLOAD_LIBVIRT_PORT}:3081 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml + upload-libvirt-xml +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lib/sched_client.rb: add some methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..db60fa2 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type: '', parameter: '', value: '') + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH lkp-tests] lib/bootstrap: delete extra space
by Liu Yinsi 26 Jan '21

26 Jan '21

before: PUB_IP=$( ip route ... after: PUB_IP=$(ip route ... Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..1127de140 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -287,7 +287,7 @@ export_ip_mac() { if has_cmd ip; then export PUB_NIC=$(ip route get 1.2.3.4 | awk '{print $5; exit}') - export PUB_IP=$( ip route get 1.2.3.4 | awk '{print $7; exit}') + export PUB_IP=$(ip route get 1.2.3.4 | awk '{print $7; exit}') elif has_cmd route; then export PUB_NIC=$(route -n | awk '/[UG][UG]/ {print $8}') export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') -- 2.23.0

1 0

[PATCH v3 compass-ci 4/5] upload-libvirt-xml/openresty.sh: start openresty in upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/openresty.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/upload-libvirt-xml/openresty.sh diff --git a/container/upload-libvirt-xml/openresty.sh b/container/upload-libvirt-xml/openresty.sh new file mode 100755 index 0000000..6215d91 --- /dev/null +++ b/container/upload-libvirt-xml/openresty.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +umask 002 +/usr/local/openresty/bin/openresty -g 'daemon off;' -- 2.23.0

1 0

[PATCH v3 compass-ci 2/5] upload-libvirt-xml/Dockerfile: add upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Add upload-libvirt-xml docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/upload-libvirt-xml/Dockerfile diff --git a/container/upload-libvirt-xml/Dockerfile b/container/upload-libvirt-xml/Dockerfile new file mode 100644 index 0000000..db48bcb --- /dev/null +++ b/container/upload-libvirt-xml/Dockerfile @@ -0,0 +1,14 @@ +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM openresty/openresty:alpine + +RUN apk add bash +RUN mkdir -p /var/log/nginx/ +RUN adduser lkp -u 1090 -g 1090 -D + +COPY ./openresty.sh /usr/local/bin/ + +CMD ["/usr/local/bin/openresty.sh"] -- 2.23.0

1 0

[PATCH compass-ci 2/4] sparrow/3-code: set default values
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] $CCI_SRC/sbin/download-rootfs: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz download rootfs needs SRV_HTTP_PORT, so set default values. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 2ba9662..4638e15 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -22,6 +22,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${RESULT_WEBDAV_PORT:=3080} : ${ES_HOST:=$server_ip} : ${ES_PORT:=9200} +: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -50,6 +51,7 @@ RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT lab: nolab ES_HOST: $ES_HOST ES_PORT: $ES_PORT +SRV_HTTP_PORT: $SRV_HTTP_PORT EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 26 Jan '21

26 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..a577b3e --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 26 Jan '21

26 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 +++++++++++++++++++++++ container/lifecycle/build | 18 +++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 33 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 78 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..d89a181 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..7e67b3c --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

3 2

[PATCH compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 3 ++- container/os-cifs/start | 3 ++- container/os-nfs/start | 6 ++++-- container/result-cifs/start | 3 ++- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..c70b318 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..6df167f 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..3b0ebb5 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -5,10 +5,12 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + echo "[ERROR] nfs module is not loaded in the Docker host's kernel (try: sudo modprobe nfs)" + exit } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + echo "[ERROR] nfsd module is not loaded in the Docker host's kernel (try: sudo modprobe nfsd)" + exit } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..3992cdd 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm result-cifs -- 2.23.0

2 2

[PATCH v2 compass-ci] lib: introduce log.rb
by Xiao Shenwei 26 Jan '21

26 Jan '21

Records the run logs of the ruby program message format: 2021-01-25 11:00:09 +0800 INFO -- SCHED_PORT: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/log.rb | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..25c92c6 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# records the run logs of the ruby program +class Log < Logger + def initialize(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + transform_msg(msg).map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end + + def transform_msg(msg) + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + return msg + end +end -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends: fix install sysbench pkg depends
by Wei Jihui 26 Jan '21

26 Jan '21

[error] Error: Unable to find a match: libmysqlclient-dev [why] openeuler has not libmysqlclient-dev: mysql-devel change it to libmariadbclient-dev Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/adaptation/openeuler | 1 - distro/depends/sysbench-dev | 2 +- distro/depends/sysbench-mysql-dev | 1 - 3 files changed, 1 insertion(+), 3 deletions(-) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 1938a8aa..9fd38b66 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -281,7 +281,6 @@ dh-python: dh-strip-nondeterminism: dh-systemd: guile-2.0: guile -libmysqlclient-dev: mysql-devel libipc-run-perl: libklibc-dev: libpython2.7: diff --git a/distro/depends/sysbench-dev b/distro/depends/sysbench-dev index 3fa67deb..f570c463 100644 --- a/distro/depends/sysbench-dev +++ b/distro/depends/sysbench-dev @@ -1,2 +1,2 @@ pkg-config -libmysqlclient-dev +libmariadbclient-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index ca5b03d8..65633b33 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,3 +1,2 @@ pkg-config -libmysqlclient-dev mysql-server -- 2.23.0

2 1

[PATCH compass-ci] container/*: replace gem sources
by Lin Jiaxin 26 Jan '21

26 Jan '21

gr "http://rubygems.org" | awk -F':' '{print $1}' | xargs sed -i 's|echo '\'':sources: \["http://rubygems.org"\]'\'' >> ~/.gemrc|gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/|g' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/assist-result/Dockerfile | 2 +- container/auto-submit/Dockerfile | 2 +- container/extract-stats/Dockerfile | 2 +- container/fluentd-base/Dockerfile | 2 +- container/git-mirror/Dockerfile | 2 +- container/mail-robot/Dockerfile | 2 +- container/monitoring/Dockerfile | 2 +- container/scheduler-dev/Dockerfile | 2 +- container/scheduler/Dockerfile | 2 +- container/webhook/Dockerfile | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..1a02f49 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -11,5 +11,5 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport json yaml elasticsearch sinatra puma terminal-table diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5804642..2a64c61 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -11,7 +11,7 @@ RUN apk update && \ apk add --no-cache git && \ apk add ruby-dev make gcc g++ util-linux -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN umask 002 && \ gem install bunny json activesupport git rest-client io-console elasticsearch diff --git a/container/extract-stats/Dockerfile b/container/extract-stats/Dockerfile index 31f4ef0..3a8c8f3 100644 --- a/container/extract-stats/Dockerfile +++ b/container/extract-stats/Dockerfile @@ -9,7 +9,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'grep' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport git json yaml elasticsearch ENV RUNTIME_DIR /c/cci/extract diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index ef671c2..08bcd9c 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -13,7 +13,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install fluentd && \ gem install fluent-plugin-rabbitmq && \ gem install fluent-plugin-elasticsearch && \ diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 8b2a50a..2eac95c 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,7 +10,7 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ apk add ruby-dev make gcc g++ openssh diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index f0aebbd..4402543 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -11,5 +11,5 @@ RUN apt-get update && \ apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install fileutils elasticsearch activesupport diff --git a/container/monitoring/Dockerfile b/container/monitoring/Dockerfile index 5ceaf4f..0a5dc22 100644 --- a/container/monitoring/Dockerfile +++ b/container/monitoring/Dockerfile @@ -8,7 +8,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client && \ gem install activesupport && \ gem install git && \ diff --git a/container/scheduler-dev/Dockerfile b/container/scheduler-dev/Dockerfile index e0c9b4a..3b68d4a 100644 --- a/container/scheduler-dev/Dockerfile +++ b/container/scheduler-dev/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER chief <taxcom(a)tom.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport bigdecimal open3 && \ gem install git json yaml && \ gem install cucumber diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index edac026..0ad813c 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -11,7 +11,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client && \ gem install activesupport && \ gem install git && \ diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile index 9299bb6..1d5ee7c 100644 --- a/container/webhook/Dockerfile +++ b/container/webhook/Dockerfile @@ -13,7 +13,7 @@ RUN apk update && \ apk add --no-cache git && \ apk add npm ruby-dev -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN npm install git-webhook-handler && \ gem install bunny -- 2.23.0

2 1

[PATCH compass-ci] sbin/es-jobs: uniform wrap style
by Lin Jiaxin 26 Jan '21

26 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/es-jobs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/es-jobs b/sbin/es-jobs index c055fe9..ff0d540 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -5,7 +5,6 @@ require 'optparse' require 'json' - require_relative '../lib/es_jobs' def parse_argv @@ -19,6 +18,7 @@ def parse_argv end items end + opt_refine = [-1] opt_fields = [] opt_stats_filter = [] -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/compare_matrixes.rb: fix TypeError
by Lu Weitao 25 Jan '21

25 Jan '21

[Why] When use compare as: sbin/compare id=crystal.120202 id=crystal.122057 met error: /home/lwt/work4/compass-ci/lib/compare_matrixes.rb:182:in `[]': no implicit conversion of String into Integer (TypeError) root cause: we call the function: def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) like: compare_matrixes(matrices_list, suite_list, options: options) the "options" will passed as "latest_jobs" [How] set nil as default value to "latest_jobs" like: def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..12c1f3e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -179,6 +179,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) end def latest_failure?(field, latest_jobs) + return true unless latest_jobs latest_jobs.any? { |job| job['stats'][field] } end @@ -203,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_matrixes.rb: fix TypeError
by Lu Weitao 25 Jan '21

25 Jan '21

[Why] When use compare as: sbin/compare id=crystal.120202 id=crystal.122057 met error: /home/lwt/work4/compass-ci/lib/compare_matrixes.rb:182:in `[]': no implicit conversion of String into Integer (TypeError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..12c1f3e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -179,6 +179,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) end def latest_failure?(field, latest_jobs) + return true unless latest_jobs latest_jobs.any? { |job| job['stats'][field] } end @@ -203,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 ++++++++++++++++++++++ container/lifecycle/build | 18 ++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 35 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 80 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..d89a181 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..35d4198 --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,35 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +#DEFAULT_CCI = '/c/cci' +#DEFAULT_LKP = '/c/lkp-tests' +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..a577b3e --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 26 ++++++++++++++++++++++ container/lifecycle/build | 18 ++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 36 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 82 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..ed5f6ee --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,26 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update + +RUN apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..5862403 --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -e CCI_SRC=#{DEFAULT_LKP} + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

3 2

[PATCH v2 lab-z9] add hosts-file of taishan200-2280-2s64p-256g--a5 which include NVMe SSD
by Zhang Dewan 25 Jan '21

25 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a5 | 25 +++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a6 | 25 +++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a5 create mode 100644 hosts/taishan200-2280-2s64p-256g--a6 diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 new file mode 100644 index 0000000..e6c11b3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981306c5 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 +mac_addr: + - 84:46:fe:26:d3:47 + - 84:46:fe:26:d3:48 + - 84:46:fe:26:d3:49 + - 84:46:fe:26:d3:4a + - ac:b3:b5:1e:b4:ab + - ac:b3:b5:1e:b4:ac + - ac:b3:b5:1e:b4:ad + - ac:b3:b5:1e:b4:ae + - ac:b3:b5:1e:ba:9b + - ac:b3:b5:1e:ba:9c + - ac:b3:b5:1e:ba:9d + - ac:b3:b5:1e:ba:9e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a6 b/hosts/taishan200-2280-2s64p-256g--a6 new file mode 100644 index 0000000..0944193 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a6 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981344d9 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000301 +mac_addr: + - 84:46:fe:89:ba:56 + - 84:46:fe:89:ba:57 + - 84:46:fe:89:ba:58 + - 84:46:fe:89:ba:59 + - ac:b3:b5:1e:b5:8b + - ac:b3:b5:1e:b5:8c + - ac:b3:b5:1e:b5:8d + - ac:b3:b5:1e:b5:8e + - ac:b3:b5:1e:b6:eb + - ac:b3:b5:1e:b6:ec + - ac:b3:b5:1e:b6:ed + - ac:b3:b5:1e:b6:ee +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.6 -- 2.23.0

1 0

[PATCH compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..d4d6ca1 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def performing_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..9a191ab --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find testboxs that are perforing tasks + # curl http://localhost:11311/performing_testbox?size=10&from=0 + get "/performing_testbox" do |env| + env.lifecycle.performing_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. [how] add SCHED_HOST SCHED_PORT on constants.rb Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ lib/sched_client.rb | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Jan '21

25 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs than runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..54affa1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/constants.rb: add SCHED_HOST SCHED_PORT field
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/compare_error_messages.rb: return true if job's field job_state is finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' field job_state is finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..9c864a3 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -13,9 +13,12 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # the later_job_id is credible. def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + later_es_result = es.query_by_id(later_job_id) + + return true if later_es_result['job_state'] == 'finished' previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') - later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') filenames_check = filenames_check(previous_result_file, later_result_file, error_id) -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: fix script hangs when no job now
by Liu Yinsi 25 Jan '21

25 Jan '21

[error] when no job now, $initrds is null, execute my-qemu.sh will hangs. lys@crystal ~/compass-ci/providers% ./my-qemu.sh hostname: vm-2p8g.lys-3917841 mac: 0a-4b-33-a0-96-69 DoneDone % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 79 100 79 0 0 19750 0 --:--:-- --:--:-- --:--:-- 19750 lys@crystal ~/compass-ci/providers/qemu% cat --help cat Copy standard input to standard output. when input is null, script will hang to waiting for input. [how] function set_options(): cat $initrds > $initrd before: execut cat command, then check after: check kernel, check initrds, if not null, execute cat command. in addition, check_option_value -> check_kernel, check_initrds, check_qemu convenient to call as needed. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 4d05964..b4e87fc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() done < $ipxe_script } -check_option_value() +check_kernel() { [ -n "$kernel" ] || { log_info "Can not find job for current hostname: $hostname." @@ -64,10 +64,16 @@ check_option_value() log_error "Can not find kernel file or kernel file is empty: $kernel." exit 1 } - +} + +check_qemu() +{ # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} +check_initrds() +{ [ -n "$initrds" ] || { log_error "The current initrds is null." exit 1 @@ -118,6 +124,8 @@ set_qemu() do command -v "$qemu" > /dev/null && break done + + check_qemu } print_message() @@ -209,8 +217,10 @@ write_logfile parse_ipxe_script +check_kernel +check_initrds + set_options -check_option_value print_message -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/compare_error_messages.rb: return true if job state = finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' state = finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..9c864a3 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -13,9 +13,12 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # the later_job_id is credible. def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + later_es_result = es.query_by_id(later_job_id) + + return true if later_es_result['job_state'] == 'finished' previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') - later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') filenames_check = filenames_check(previous_result_file, later_result_file, error_id) -- 2.23.0

2 2

[PATCH v1 compass-ci] lib: introduce log.rb
by Xiao Shenwei 25 Jan '21

25 Jan '21

Records the run logs of the ruby program message format: 2021-01-25 11:00:09 +0800 INFO -- SCHED_PORT: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/log.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..762c0b2 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# records the run logs of the ruby program +class Log < Logger + def initialize(filename) + File.delete(filename) if FileTest.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + transform_msg(msg).map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end + + def transform_msg(msg) + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + return msg + end +end -- 2.23.0

2 3

[PATCH compass-ci] lib/compare_error_messages.rb: return true if job state = finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' state = finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..978d6ba 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -14,6 +14,8 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + return true if es.query_by_id(later_job_id)['job_state'] == 'finished' + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') -- 2.23.0

2 1

[PATCH lab-z9] hosts: replace field rootfs_partition to rootfs_disk of all hosts file
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] 1. The current value of rootfs_partition field is actually disk, not the partition. 2. We now need auto install iso to the specific disk of testbox, so in order to avoid conflict with rootfs_partition in lkp-tests framework, we use rootfs_disk instead of rootfs_partition. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a11 | 2 +- hosts/taishan200-2280-2s48p-256g--a12 | 2 +- hosts/taishan200-2280-2s48p-256g--a13 | 2 +- hosts/taishan200-2280-2s48p-256g--a14 | 2 +- hosts/taishan200-2280-2s48p-256g--a15 | 2 +- hosts/taishan200-2280-2s48p-256g--a17 | 2 +- hosts/taishan200-2280-2s48p-256g--a18 | 2 +- hosts/taishan200-2280-2s48p-256g--a19 | 2 +- hosts/taishan200-2280-2s48p-256g--a21 | 2 +- hosts/taishan200-2280-2s48p-256g--a22 | 2 +- hosts/taishan200-2280-2s48p-256g--a24 | 2 +- hosts/taishan200-2280-2s48p-256g--a25 | 2 +- hosts/taishan200-2280-2s48p-256g--a26 | 2 +- hosts/taishan200-2280-2s48p-256g--a27 | 2 +- hosts/taishan200-2280-2s48p-256g--a28 | 2 +- hosts/taishan200-2280-2s48p-256g--a29 | 2 +- hosts/taishan200-2280-2s48p-256g--a30 | 2 +- hosts/taishan200-2280-2s48p-256g--a31 | 2 +- hosts/taishan200-2280-2s48p-256g--a32 | 2 +- hosts/taishan200-2280-2s48p-256g--a33 | 2 +- hosts/taishan200-2280-2s48p-256g--a34 | 2 +- hosts/taishan200-2280-2s48p-256g--a52 | 2 +- hosts/taishan200-2280-2s48p-256g--a53 | 2 +- hosts/taishan200-2280-2s48p-256g--a60 | 2 +- hosts/taishan200-2280-2s48p-256g--a61 | 2 +- hosts/taishan200-2280-2s48p-256g--a62 | 2 +- hosts/taishan200-2280-2s48p-256g--a66 | 2 +- hosts/taishan200-2280-2s48p-256g--a67 | 2 +- hosts/taishan200-2280-2s48p-256g--a68 | 2 +- hosts/taishan200-2280-2s48p-256g--a69 | 2 +- hosts/taishan200-2280-2s48p-256g--a70 | 2 +- hosts/taishan200-2280-2s64p-256g--a10 | 2 +- hosts/taishan200-2280-2s64p-256g--a3 | 2 +- hosts/taishan200-2280-2s64p-256g--a36 | 2 +- hosts/taishan200-2280-2s64p-256g--a4 | 2 +- hosts/taishan200-2280-2s64p-256g--a40 | 2 +- hosts/taishan200-2280-2s64p-256g--a41 | 2 +- hosts/taishan200-2280-2s64p-256g--a42 | 2 +- hosts/taishan200-2280-2s64p-256g--a8 | 2 +- hosts/taishan200-2280-2s64p-256g--a9 | 2 +- 40 files changed, 40 insertions(+), 40 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a11 b/hosts/taishan200-2280-2s48p-256g--a11 index 619dc063cf0f..412154f8c8e8 100644 --- a/hosts/taishan200-2280-2s48p-256g--a11 +++ b/hosts/taishan200-2280-2s48p-256g--a11 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8891431-part1 - /dev/disk/by-id/scsi-350000399b8890e09 - /dev/disk/by-id/scsi-350000399b8890cd1 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8727929 mac_addr: - cc:05:77:ab:97:62 diff --git a/hosts/taishan200-2280-2s48p-256g--a12 b/hosts/taishan200-2280-2s48p-256g--a12 index ce014c133bb6..3baa8f187ba3 100644 --- a/hosts/taishan200-2280-2s48p-256g--a12 +++ b/hosts/taishan200-2280-2s48p-256g--a12 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c500c276c347 - /dev/disk/by-id/scsi-35000c500c276eb87 - /dev/disk/by-id/scsi-35000c500c276b6ff -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500c276f47b mac_addr: - 44:67:47:97:22:37 diff --git a/hosts/taishan200-2280-2s48p-256g--a13 b/hosts/taishan200-2280-2s48p-256g--a13 index 468c175a3111..4b9548fa032e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a13 +++ b/hosts/taishan200-2280-2s48p-256g--a13 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c500c2799bb3 - /dev/disk/by-id/scsi-35000c500c278b18b - /dev/disk/by-id/scsi-35000c500c279f96b -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500c2789b4b mac_addr: - 44:67:47:97:2b:bf diff --git a/hosts/taishan200-2280-2s48p-256g--a14 b/hosts/taishan200-2280-2s48p-256g--a14 index 24ea9dc8daf5..b96e79c9029a 100644 --- a/hosts/taishan200-2280-2s48p-256g--a14 +++ b/hosts/taishan200-2280-2s48p-256g--a14 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4475 - /dev/disk/by-id/scsi-350000399b88b4371 - /dev/disk/by-id/scsi-350000399b88b432d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88b4531 mac_addr: - 44:67:47:c9:e9:c2 diff --git a/hosts/taishan200-2280-2s48p-256g--a15 b/hosts/taishan200-2280-2s48p-256g--a15 index 9c28258def58..a1d63ed4fcc5 100644 --- a/hosts/taishan200-2280-2s48p-256g--a15 +++ b/hosts/taishan200-2280-2s48p-256g--a15 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b841fff5 - /dev/disk/by-id/scsi-350000399b841fcb9 - /dev/disk/by-id/scsi-350000399b8420a0d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b841ff55 mac_addr: - 44:67:47:85:d4:9e diff --git a/hosts/taishan200-2280-2s48p-256g--a17 b/hosts/taishan200-2280-2s48p-256g--a17 index 16b065db0f66..9437ac9ea397 100644 --- a/hosts/taishan200-2280-2s48p-256g--a17 +++ b/hosts/taishan200-2280-2s48p-256g--a17 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b44cd - /dev/disk/by-id/scsi-350000399b88b4541 - /dev/disk/by-id/scsi-350000399b88b4495 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88b48d1 mac_addr: - 44:67:47:c9:ea:1c diff --git a/hosts/taishan200-2280-2s48p-256g--a18 b/hosts/taishan200-2280-2s48p-256g--a18 index 9d1ec017438a..0567352ac36b 100644 --- a/hosts/taishan200-2280-2s48p-256g--a18 +++ b/hosts/taishan200-2280-2s48p-256g--a18 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b891b4d1 - /dev/disk/by-id/scsi-350000399b8890d59 - /dev/disk/by-id/scsi-350000399b8890f65 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b891b3c5 mac_addr: - 84:46:fe:3d:a7:e9 diff --git a/hosts/taishan200-2280-2s48p-256g--a19 b/hosts/taishan200-2280-2s48p-256g--a19 index 217eaa9ec067..7e86603af883 100644 --- a/hosts/taishan200-2280-2s48p-256g--a19 +++ b/hosts/taishan200-2280-2s48p-256g--a19 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8923939 - /dev/disk/by-id/scsi-350000399b8420b2d - /dev/disk/by-id/scsi-350000399b891b5c5 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b841f769 mac_addr: - 44:67:47:85:d5:48 diff --git a/hosts/taishan200-2280-2s48p-256g--a21 b/hosts/taishan200-2280-2s48p-256g--a21 index 738738d134cb..72fb511ae1bb 100644 --- a/hosts/taishan200-2280-2s48p-256g--a21 +++ b/hosts/taishan200-2280-2s48p-256g--a21 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf2d5 - /dev/disk/by-id/scsi-350000399b88bf5a5 - /dev/disk/by-id/scsi-350000399b88bf279 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bf24d mac_addr: - 44:67:47:86:1b:e0 diff --git a/hosts/taishan200-2280-2s48p-256g--a22 b/hosts/taishan200-2280-2s48p-256g--a22 index bb29d9432b52..73a2799006a3 100644 --- a/hosts/taishan200-2280-2s48p-256g--a22 +++ b/hosts/taishan200-2280-2s48p-256g--a22 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919069 - /dev/disk/by-id/scsi-350000399b8918c15 - /dev/disk/by-id/scsi-350000399b8918c79 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8918cb9 mac_addr: - 44:67:47:d7:6d:14 diff --git a/hosts/taishan200-2280-2s48p-256g--a24 b/hosts/taishan200-2280-2s48p-256g--a24 index 60ea1d2a1588..8f78184d4da0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a24 +++ b/hosts/taishan200-2280-2s48p-256g--a24 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917d59 - /dev/disk/by-id/scsi-350000399b89185d9 - /dev/disk/by-id/scsi-350000399b891790d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8917ef1 mac_addr: - 44:67:47:c9:d8:f4 diff --git a/hosts/taishan200-2280-2s48p-256g--a25 b/hosts/taishan200-2280-2s48p-256g--a25 index 916513e68f0e..d8f721c58888 100644 --- a/hosts/taishan200-2280-2s48p-256g--a25 +++ b/hosts/taishan200-2280-2s48p-256g--a25 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4499 - /dev/disk/by-id/scsi-350000399b88b4455 - /dev/disk/by-id/scsi-350000399b88b442d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88b44f9 mac_addr: - 44:67:47:c9:ea:08 diff --git a/hosts/taishan200-2280-2s48p-256g--a26 b/hosts/taishan200-2280-2s48p-256g--a26 index 4a5a0f350bec..210e15d2c724 100644 --- a/hosts/taishan200-2280-2s48p-256g--a26 +++ b/hosts/taishan200-2280-2s48p-256g--a26 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf131 - /dev/disk/by-id/scsi-350000399b88bfd8d - /dev/disk/by-id/scsi-350000399b88bfe55 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bef31 mac_addr: - 44:67:47:c9:e9:40 diff --git a/hosts/taishan200-2280-2s48p-256g--a27 b/hosts/taishan200-2280-2s48p-256g--a27 index 129589cde687..5d5559e35597 100644 --- a/hosts/taishan200-2280-2s48p-256g--a27 +++ b/hosts/taishan200-2280-2s48p-256g--a27 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918ea9 - /dev/disk/by-id/scsi-350000399b8918bad - /dev/disk/by-id/scsi-350000399b8918d05 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8919415 mac_addr: - 44:67:47:d7:6d:3c diff --git a/hosts/taishan200-2280-2s48p-256g--a28 b/hosts/taishan200-2280-2s48p-256g--a28 index f8a77f8408b9..2e32147db8ff 100644 --- a/hosts/taishan200-2280-2s48p-256g--a28 +++ b/hosts/taishan200-2280-2s48p-256g--a28 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c500cdf1eb07 - /dev/disk/by-id/scsi-35000c500cdef4c6f - /dev/disk/by-id/scsi-35000c500cdefac2b -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500cdefa75f mac_addr: - 5c:e8:83:31:c8:20 diff --git a/hosts/taishan200-2280-2s48p-256g--a29 b/hosts/taishan200-2280-2s48p-256g--a29 index 37814cc0e043..fff6a831f2aa 100644 --- a/hosts/taishan200-2280-2s48p-256g--a29 +++ b/hosts/taishan200-2280-2s48p-256g--a29 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918425 - /dev/disk/by-id/scsi-350000399b891841d - /dev/disk/by-id/scsi-350000399b8917859 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8917a51 mac_addr: - 44:67:47:c9:d8:ea diff --git a/hosts/taishan200-2280-2s48p-256g--a30 b/hosts/taishan200-2280-2s48p-256g--a30 index f722edbd7795..713d7af7017d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a30 +++ b/hosts/taishan200-2280-2s48p-256g--a30 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b883b881 - /dev/disk/by-id/scsi-350000399b88bf83d - /dev/disk/by-id/scsi-350000399b883bbbd -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bf48d mac_addr: - 44:67:47:86:1b:cc diff --git a/hosts/taishan200-2280-2s48p-256g--a31 b/hosts/taishan200-2280-2s48p-256g--a31 index 7d6c67cc8400..c112fe2c6235 100644 --- a/hosts/taishan200-2280-2s48p-256g--a31 +++ b/hosts/taishan200-2280-2s48p-256g--a31 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b883b929 - /dev/disk/by-id/scsi-350000399b883bbe1 - /dev/disk/by-id/scsi-350000399b883c059 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b883bc9d mac_addr: - 44:67:47:86:1b:ae diff --git a/hosts/taishan200-2280-2s48p-256g--a32 b/hosts/taishan200-2280-2s48p-256g--a32 index fb17e38aebb1..212db948e368 100644 --- a/hosts/taishan200-2280-2s48p-256g--a32 +++ b/hosts/taishan200-2280-2s48p-256g--a32 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf141 - /dev/disk/by-id/scsi-350000399b88bee25 - /dev/disk/by-id/scsi-350000399b88bea59 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88be1d9 mac_addr: - 44:67:47:c9:db:38 diff --git a/hosts/taishan200-2280-2s48p-256g--a33 b/hosts/taishan200-2280-2s48p-256g--a33 index a3a6b0306ad4..96bb5a4b5d1d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a33 +++ b/hosts/taishan200-2280-2s48p-256g--a33 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919555 - /dev/disk/by-id/scsi-350000399b8918c4d - /dev/disk/by-id/scsi-350000399b8919c3d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8919389 mac_addr: - 44:67:47:d7:6c:c4 diff --git a/hosts/taishan200-2280-2s48p-256g--a34 b/hosts/taishan200-2280-2s48p-256g--a34 index 5a15949fe123..727f870fb7e6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a34 +++ b/hosts/taishan200-2280-2s48p-256g--a34 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88be219 - /dev/disk/by-id/scsi-350000399b88bdf71 - /dev/disk/by-id/scsi-350000399b88bdb49 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b891784d mac_addr: - 44:67:47:c9:d9:26 diff --git a/hosts/taishan200-2280-2s48p-256g--a52 b/hosts/taishan200-2280-2s48p-256g--a52 index 2bd70962a573..0deb149ef6fa 100644 --- a/hosts/taishan200-2280-2s48p-256g--a52 +++ b/hosts/taishan200-2280-2s48p-256g--a52 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c500cdefd89b - /dev/disk/by-id/scsi-35000c500cdefac67 - /dev/disk/by-id/scsi-35000c500cdedee6f -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500cdedfdaf mac_addr: - 5c:e8:83:31:c8:d4 diff --git a/hosts/taishan200-2280-2s48p-256g--a53 b/hosts/taishan200-2280-2s48p-256g--a53 index bf0894b46f99..ca075737a4e6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a53 +++ b/hosts/taishan200-2280-2s48p-256g--a53 @@ -5,7 +5,7 @@ memory: 256G nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd6aa4e3 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500bd681acf mac_addr: - 5c:e8:83:73:25:c9 diff --git a/hosts/taishan200-2280-2s48p-256g--a60 b/hosts/taishan200-2280-2s48p-256g--a60 index 2255f2f52478..13612be3785d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a60 +++ b/hosts/taishan200-2280-2s48p-256g--a60 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd67640b - /dev/disk/by-id/scsi-35000c500bd6707a7 - /dev/disk/by-id/scsi-35000c500bd680dc3 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500bd683a73 mac_addr: - e0:00:84:2b:51:7d diff --git a/hosts/taishan200-2280-2s48p-256g--a61 b/hosts/taishan200-2280-2s48p-256g--a61 index 73079cbebb0a..1619d17f99e9 100644 --- a/hosts/taishan200-2280-2s48p-256g--a61 +++ b/hosts/taishan200-2280-2s48p-256g--a61 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399886a7869 - /dev/disk/by-id/scsi-350000399886acc5d - /dev/disk/by-id/scsi-350000399886b6861 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399886ac70d mac_addr: - 44:67:47:c9:e4:b8 diff --git a/hosts/taishan200-2280-2s48p-256g--a62 b/hosts/taishan200-2280-2s48p-256g--a62 index abaac6e80c02..f1d72af845d0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a62 +++ b/hosts/taishan200-2280-2s48p-256g--a62 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b892d131 - /dev/disk/by-id/scsi-350000399b892d619 - /dev/disk/by-id/scsi-350000399b892dc1d -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b892dbb5 mac_addr: - 44:67:47:c9:e3:3a diff --git a/hosts/taishan200-2280-2s48p-256g--a66 b/hosts/taishan200-2280-2s48p-256g--a66 index fce93d166425..24d9f04e52ec 100644 --- a/hosts/taishan200-2280-2s48p-256g--a66 +++ b/hosts/taishan200-2280-2s48p-256g--a66 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-35000c5009a671773 - /dev/disk/by-id/scsi-35000c5009a690c53 - /dev/disk/by-id/scsi-35000c5009a6713fb -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c5009a65963f mac_addr: - 5c:e8:83:31:c8:8e diff --git a/hosts/taishan200-2280-2s48p-256g--a67 b/hosts/taishan200-2280-2s48p-256g--a67 index 6b326099609c..35bf63983e71 100644 --- a/hosts/taishan200-2280-2s48p-256g--a67 +++ b/hosts/taishan200-2280-2s48p-256g--a67 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88aed21 - /dev/disk/by-id/scsi-350000399b88af325 - /dev/disk/by-id/scsi-350000399b88af699 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88af421 mac_addr: - 44:67:47:c9:e2:2c diff --git a/hosts/taishan200-2280-2s48p-256g--a68 b/hosts/taishan200-2280-2s48p-256g--a68 index 4bdd8f4fa83b..7c5091311f76 100644 --- a/hosts/taishan200-2280-2s48p-256g--a68 +++ b/hosts/taishan200-2280-2s48p-256g--a68 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b8909f81 - /dev/disk/by-id/scsi-350000399b8909f79 - /dev/disk/by-id/scsi-350000399b8909605 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8909b85 mac_addr: - 84:46:fe:89:b6:e6 diff --git a/hosts/taishan200-2280-2s48p-256g--a69 b/hosts/taishan200-2280-2s48p-256g--a69 index 3d3be8042fb2..c796b53bb529 100644 --- a/hosts/taishan200-2280-2s48p-256g--a69 +++ b/hosts/taishan200-2280-2s48p-256g--a69 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b88be92d - /dev/disk/by-id/scsi-350000399b88befc5 - /dev/disk/by-id/scsi-350000399b88be701 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bea5d mac_addr: - 44:67:47:c9:d9:4e diff --git a/hosts/taishan200-2280-2s48p-256g--a70 b/hosts/taishan200-2280-2s48p-256g--a70 index 89c3c772fa7b..6064b87f718c 100644 --- a/hosts/taishan200-2280-2s48p-256g--a70 +++ b/hosts/taishan200-2280-2s48p-256g--a70 @@ -7,7 +7,7 @@ hdd_partitions: - /dev/disk/by-id/scsi-350000399b890931d - /dev/disk/by-id/scsi-350000399b8909d79 - /dev/disk/by-id/scsi-350000399b8909ac5 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8909c99 mac_addr: - 84:46:fe:89:b7:22 diff --git a/hosts/taishan200-2280-2s64p-256g--a10 b/hosts/taishan200-2280-2s64p-256g--a10 index cdb32f54343f..c6e6b56dc6f1 100644 --- a/hosts/taishan200-2280-2s64p-256g--a10 +++ b/hosts/taishan200-2280-2s64p-256g--a10 @@ -3,7 +3,7 @@ nr_node: 4 nr_cpu: 128 memory: 256G nr_ssd_partitions: 1 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399c8030fd5 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44333 diff --git a/hosts/taishan200-2280-2s64p-256g--a3 b/hosts/taishan200-2280-2s64p-256g--a3 index 8f34f3f8bdec..1c7a2e9fc4f2 100644 --- a/hosts/taishan200-2280-2s64p-256g--a3 +++ b/hosts/taishan200-2280-2s64p-256g--a3 @@ -7,7 +7,7 @@ ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399981347d5 mac_addr: - 68:4a:ae:ca:e8:ca diff --git a/hosts/taishan200-2280-2s64p-256g--a36 b/hosts/taishan200-2280-2s64p-256g--a36 index 97571449df9c..1b0bf294956d 100644 --- a/hosts/taishan200-2280-2s64p-256g--a36 +++ b/hosts/taishan200-2280-2s64p-256g--a36 @@ -3,7 +3,7 @@ nr_node: 4 nr_cpu: 128 memory: 256G nr_ssd_partitions: 1 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-3500003999813487d ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB42949-part1 diff --git a/hosts/taishan200-2280-2s64p-256g--a4 b/hosts/taishan200-2280-2s64p-256g--a4 index bcfaea74799d..b3a4ad7b2da0 100644 --- a/hosts/taishan200-2280-2s64p-256g--a4 +++ b/hosts/taishan200-2280-2s64p-256g--a4 @@ -7,7 +7,7 @@ ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part1 - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part2 - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part3 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399c802cfb9 mac_addr: - 68:4a:ae:ca:e8:a2 diff --git a/hosts/taishan200-2280-2s64p-256g--a40 b/hosts/taishan200-2280-2s64p-256g--a40 index d9a993725dbc..f30f7fb4f8ea 100644 --- a/hosts/taishan200-2280-2s64p-256g--a40 +++ b/hosts/taishan200-2280-2s64p-256g--a40 @@ -5,7 +5,7 @@ memory: 256G nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-3500003999812ed81 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399c8020115 mac_addr: - 84:46:fe:89:be:8e diff --git a/hosts/taishan200-2280-2s64p-256g--a41 b/hosts/taishan200-2280-2s64p-256g--a41 index 91531fded637..ab2073edd46c 100644 --- a/hosts/taishan200-2280-2s64p-256g--a41 +++ b/hosts/taishan200-2280-2s64p-256g--a41 @@ -5,7 +5,7 @@ memory: 256G nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919495 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000039998130a85 mac_addr: - 84:46:fe:89:bc:4a diff --git a/hosts/taishan200-2280-2s64p-256g--a42 b/hosts/taishan200-2280-2s64p-256g--a42 index ed1f188b00f3..ecaccae5bda0 100644 --- a/hosts/taishan200-2280-2s64p-256g--a42 +++ b/hosts/taishan200-2280-2s64p-256g--a42 @@ -5,7 +5,7 @@ memory: 256G nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918a11 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-350000399c802ee75 mac_addr: - cc:05:77:fe:d8:64 diff --git a/hosts/taishan200-2280-2s64p-256g--a8 b/hosts/taishan200-2280-2s64p-256g--a8 index e515306aba96..8a02f712cd31 100644 --- a/hosts/taishan200-2280-2s64p-256g--a8 +++ b/hosts/taishan200-2280-2s64p-256g--a8 @@ -2,7 +2,7 @@ arch: aarch64 nr_node: 4 nr_cpu: 128 memory: 256G -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43325 mac_addr: - 84:46:fe:26:12:f5 diff --git a/hosts/taishan200-2280-2s64p-256g--a9 b/hosts/taishan200-2280-2s64p-256g--a9 index ce3aa0a99073..fd8473fb4076 100644 --- a/hosts/taishan200-2280-2s64p-256g--a9 +++ b/hosts/taishan200-2280-2s64p-256g--a9 @@ -5,7 +5,7 @@ memory: 256G nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd6a1347 -rootfs_partition: +rootfs_disk: - /dev/disk/by-id/scsi-35000c500bd62acb3 mac_addr: - cc:05:77:fe:d4:0e -- 2.23.0

1 0

[PATCH v2 lkp-tests] iso2rootfs: fix domain name cannot be resolved problem
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] We have added some nameservers to /etc/resolv.conf of rootfs, but this file will be overwritten when NetworkManager service starts. [How] Add config option `dns=none` to /etc/NetworkManager/NetworkManager.conf. [Reference] dns in NetworkManager.conf Set the DNS (resolv.conf) processing mode. default: The default if the key is not specified. NetworkManager will update resolv.conf to reflect the nameservers provided by currently active connections. dnsmasq: NetworkManager will run dnsmasq as a local caching nameserver, using a "split DNS" configuration if you are connected to a VPN, and then update resolv.conf to point to the local nameserver. none: NetworkManager will not modify resolv.conf. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index cb077fa28b0f..65a1935b7f49 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -358,6 +358,10 @@ run_qcow2rootfs() config_dns_resolver() { echo "${config_rootfs}" | grep -qw 'dns' || return 0 + + grep -qs "^dns=none" $config_file || + sed -i '/^\[main\]/adns=none' "${ROOTFS_DES_DIR}/etc/NetworkManager/NetworkManager.conf" + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" nameserver 114.114.114.114 nameserver 8.8.8.8 -- 2.23.0

1 0

[PATCH lkp-tests] iso2rootfs: fix domain name cannot be resolved problem
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] We have added some nameservers to /etc/resolv.conf of rootfs, but this file will be overwritten when NetworkManager service starts. [How] Add config option `dns=none` to /etc/NetworkManager/NetworkManager.conf. [Reference] dns in NetworkManager.conf Set the DNS (resolv.conf) processing mode. default: The default if the key is not specified. NetworkManager will update resolv.conf to reflect the nameservers provided by currently active connections. dnsmasq: NetworkManager will run dnsmasq as a local caching nameserver, using a "split DNS" configuration if you are connected to a VPN, and then update resolv.conf to point to the local nameserver. none: NetworkManager will not modify resolv.conf. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index cb077fa28b0f..cde1d12e4cd3 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -358,6 +358,9 @@ run_qcow2rootfs() config_dns_resolver() { echo "${config_rootfs}" | grep -qw 'dns' || return 0 + + grep "^dns=none" $config_file || sed -i '/^\[main\]/adns=none' "${ROOTFS_DES_DIR}/etc/NetworkManager/NetworkManager.conf" + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" nameserver 114.114.114.114 nameserver 8.8.8.8 -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/matrix2.rb support merge message|log line to stats
by Lu Weitao 25 Jan '21

25 Jan '21

[Example] { "iperf.tcp.sender.bps": 34804801216.197174, "iperf.tcp.receiver.bps": 34804762215.18231, "iperf.run.message": "error message xxx", "iperf.run.log": "error log xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..5920d62 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -10,6 +10,8 @@ require "#{LKP_SRC}/lib/matrix" require_relative './params_group.rb' def set_pre_value(item, value, sample_size) + return value if value.is_a?(String) + if value.size == 1 value[0] elsif independent_counter? item -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/dump-stats: keep message|log line
by Lu Weitao 25 Jan '21

25 Jan '21

[Example] { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ], "iperf.run.message": "error message xxx", "iperf.run.log": "error log xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/dump-stat | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/sbin/dump-stat b/sbin/dump-stat index b5abf5a9b..6e10e432d 100755 --- a/sbin/dump-stat +++ b/sbin/dump-stat @@ -59,6 +59,13 @@ while (line = STDIN.gets) exit 1 end + + # keep message | log line which key end with .message|.log + if k.end_with?('.message', '.log') + result[k] = v + next + end + # only number is valid unless v.numeric? invalid_records.push record_index @@ -84,6 +91,8 @@ min_cols_stat = '' max_cols_stat = '' zero_stats = [] result.each do |key, val| + next if val.is_a?(String) + if max_cols < val.size max_cols = val.size max_cols_stat = key -- 2.23.0

1 0

[PATCH v2 lkp-tests] params/timeout: convert timeout into seconds
by Wu Zhende 25 Jan '21

25 Jan '21

It's hard to process the "timeout" in different units in the code, and convert them into integers in seconds. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- params/timeout | 1 + 1 file changed, 1 insertion(+) create mode 120000 params/timeout diff --git a/params/timeout b/params/timeout new file mode 120000 index 000000000..9e1976b33 --- /dev/null +++ b/params/timeout @@ -0,0 +1 @@ +runtime \ No newline at end of file -- 2.23.0

1 0

[PATCH lkp-tests 2/2] params/timeout: convert timeout into seconds
by Wu Zhende 25 Jan '21

25 Jan '21

It's hard to process the "timeout" in different units in the code, and convert them into integers in seconds. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- params/timeout | 1 + 1 file changed, 1 insertion(+) create mode 120000 params/timeout diff --git a/params/timeout b/params/timeout new file mode 120000 index 000000000..9e1976b33 --- /dev/null +++ b/params/timeout @@ -0,0 +1 @@ +runtime \ No newline at end of file -- 2.23.0

1 0

[PATCH lkp-tests 1/2] sbin/submit: add submit_id
by Wu Zhende 25 Jan '21

25 Jan '21

A globally "submit_id" is automatically generated for each submit command and automatically added to jobs. Indicates the jobs of the same batch submitted by the command. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a92556fff..5273896b6 100755 --- a/sbin/submit +++ b/sbin/submit @@ -103,6 +103,9 @@ def find_jobfile(jobfile) exit 1 end +submit_id = %x(uuidgen).chomp +puts "submit_id=#{submit_id}" + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -113,6 +116,7 @@ ARGV.each do |jobfile| jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] + jobs['submit_id'] = submit_id jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

1 0

[PATCH compass-ci] taskqueue: add start dependent container: redis
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] The container taskqueue start depends on redis, so add redis to taskqueue/start-depends. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/taskqueue/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/taskqueue/start-depends diff --git a/container/taskqueue/start-depends b/container/taskqueue/start-depends new file mode 100755 index 000000000000..7800f0fad3ff --- /dev/null +++ b/container/taskqueue/start-depends @@ -0,0 +1 @@ +redis -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/dump-stats: keep message line
by Lu Weitao 25 Jan '21

25 Jan '21

[Example] { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ], "iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/dump-stat | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sbin/dump-stat b/sbin/dump-stat index b5abf5a9b..5d565e36a 100755 --- a/sbin/dump-stat +++ b/sbin/dump-stat @@ -59,6 +59,12 @@ while (line = STDIN.gets) exit 1 end + + if k.end_with?('message') + result[k] = v + next + end + # only number is valid unless v.numeric? invalid_records.push record_index @@ -84,6 +90,8 @@ min_cols_stat = '' max_cols_stat = '' zero_stats = [] result.each do |key, val| + next if val.is_a?(String) + if max_cols < val.size max_cols = val.size max_cols_stat = key -- 2.23.0

2 2

[PATCH v4 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 25 Jan '21

25 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

2 2