- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] providers/libvirt: split domain.xml to some common templates
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] libvirt can boot a vm based on a xml file named domain.xml. it contains many elements like os, device, cpu. to meet a requirements of a single module or a combination of multiple modules. split the domain.xml may be better. here are some scenarios: case 1: the user provides a complete domain.xml file case 2: the user only provides a cpu.xml template case 3: the user provides cpu.xml and disk.xml or other templates Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 5 +++++ providers/libvirt/templates/cpu.xml | 3 +++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/options.yaml | 23 +++++++++++++++++++++++ providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 14 files changed, 79 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/options.yaml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..12d0ef8 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,5 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..1f95dc6 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,3 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/options.yaml b/providers/libvirt/templates/options.yaml new file mode 100644 index 0000000..c41ee29 --- /dev/null +++ b/providers/libvirt/templates/options.yaml @@ -0,0 +1,23 @@ +# Domain option: +# use templates/domain.xml if user not specified. +domain: domain + +# Default options: +# these elements must be replaced for each domain.xml. +default: + - name + - os + - memory + - serial + - interface + - active + - emulator + +# Replaceable options: +# these elements can be replaced by user configurations. +replaceable: + - devices + - cpu + - seclabel + - disk + - clock diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..3ac880d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

review star
by Wu Fengguang 01 Feb '21

01 Feb '21

Xueliang is our review star again, thanks Xueliang! Review email records in the last 7 days total review paragraph point people 53 23 77 41 Wu Fengguang 89 32 38 32 Cao Xueliang 80 25 32 19 Wei Jihui 66 18 34 7 Xiao Shenwei 25 10 18 6 Li Yuanchao 24 4 12 6 Yu Chuan 49 7 15 6 Luan Shengde 21 7 6 5 Cui Yili 29 10 13 5 Liu Shaofei 22 5 9 3 Lin Jiaxin 34 5 7 3 Wang Yong 12 2 2 2 Bai Jing 22 2 3 1 Lu Weitao 4 1 1 1 Xu Xijian 14 1 3 1 Li Ping 96 4 10 1 Liu Yinsi 46 3 4 1 Wu Zhende 1 0 0 0 Zhang Yale 3 0 0 0 Ren Wen 19 1 2 0 Zhang Dewan 18 2 3 0 Wang Chenglong

1 0

git count
by Wu Fengguang 01 Feb '21

01 Feb '21

Wu Zhende is our code star, thanks Wu Zhende! 12 Wu Zhende 10 Liu Yinsi 9 Xiao Shenwei 7 Cao Xueliang 6 Wei Jihui 6 Yu Chuan 5 Li Yuanchao 4 Lu Weitao 4 Wu Fengguang 3 Bai Jing 3 Wang Chenglong 3 Lin Jiaxin 3 Zhang Dewan 2 Xu Xijian 2 Liu Shaofei 2 Li Ping 1 Wang Yong 1 Luan Shengde 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Cui Yili 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

[PATCH compass-ci] git_mirror.rb: fix that upstream-repos reload failed
by Li Yuanchao 01 Feb '21

01 Feb '21

As the file list get by git command will have a '\n' at the end of each line, the method can't find the file with the wrong name. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 13cb7c0..cc5c7f0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -268,6 +268,7 @@ class MirrorMain file_list.each_line do |file| next if File.basename(file) == '.ignore' + file = file.chomp repo_dir = "#{REPO_DIR}/#{file}" load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) end -- 2.23.0

1 0

[PATCH lab-z9] add host-file of a35 a37 a39
by Zhang Dewan 01 Feb '21

01 Feb '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a35 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a37 | 27 +++++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 +++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s64p-256g--a37 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..5d9d500 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,19 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a37 b/hosts/taishan200-2280-2s64p-256g--a37 new file mode 100644 index 0000000..657707a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a37 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399c802ef2d +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part3 +mac_addr: + - 84:46:fe:89:b9:66 + - 84:46:fe:89:b9:67 + - 84:46:fe:89:b9:68 + - 84:46:fe:89:b9:69 + - ac:b3:b5:1e:d5:eb + - ac:b3:b5:1e:d5:ec + - ac:b3:b5:1e:d5:ed + - ac:b3:b5:1e:d5:ee + - ac:b3:b5:1e:d6:3b + - ac:b3:b5:1e:d6:3c + - ac:b3:b5:1e:d6:3d + - ac:b3:b5:1e:d6:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.2 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

2 4

[PATCH v2 lkp-tests 2/2] sbin/install-dependencies: replace gem sources
by Liu Yinsi 01 Feb '21

01 Feb '21

Encapsulating functions install_gem_pkg for easy gem sources replacement Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index c16385e0f..012c45499 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -13,7 +13,6 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket git activesupport rest-client ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -21,7 +20,6 @@ linux_dep() else sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp fi - sudo gem install -f git activesupport rest-client faye-websocket ;; *) echo "$ID not support! please install dependencies manually." && exit 1 @@ -34,7 +32,12 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - sudo gem install -f git activesupport rest-client +} + +install_gem_pkg() +{ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + sudo gem install -f git activesupport rest-client faye-websocket } run() @@ -46,6 +49,8 @@ run() else echo "$DISTRO not supported!" && exit 1 fi + + install_gem_pkg } set_env() -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/install-dependencies: exit if conditions are not met
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] auto install dependencies must meet conditions, if not, exit script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index f2a9155f6..c16385e0f 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -24,7 +24,7 @@ linux_dep() sudo gem install -f git activesupport rest-client faye-websocket ;; *) - echo "$ID not support! please install dependencies manually." + echo "$ID not support! please install dependencies manually." && exit 1 ;; esac } @@ -44,7 +44,7 @@ run() elif is_system "Darwin"; then mac_dep else - echo "$DISTRO not supported!" + echo "$DISTRO not supported!" && exit 1 fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete fields that need to be reloaded for reproduction task
by Li Ping 01 Feb '21

01 Feb '21

[why] 1. load job from result's job yaml has key "my_email", like my_email: compass-ci-robot(a)qq.com my_name: compass-ci-robot lab: z9 2. load job from result's job yaml has key "SCHED_HOST" which from /etc/compass-ci/defaults/*.yaml, like SCHED_HOST: 172.168.131.2 SCHED_PORT: 3000 each of these fields needs to be reset when submit job.yaml for reprodution task. however, these fields already exist in yaml file, can not to be reload, so delete these fields when submit reproduction job. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 640b25a5f..867c3d193 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -258,6 +258,13 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + if hash.has_key?('id') + hash.delete_if { |key, _| key == "SCHED_HOST" } + hash.delete_if { |key, _| key == "SCHED_PORT" } + hash.delete_if { |key, _| key == "my_name" } + hash.delete_if { |key, _| key == "my_email" } + hash.delete_if { |key, _| key == "lab" } + end revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? -- 2.23.0

3 4

[PATCH v2 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..bcce612 --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline: #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4