- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc/write-PKGBUILD: fix documentation errors
by Lin Jiaxin 03 Feb '21

03 Feb '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/write-PKGBUILD.en.md | 5 +++-- doc/manual/write-PKGBUILD.zh.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/manual/write-PKGBUILD.en.md b/doc/manual/write-PKGBUILD.en.md index c0c25e9..c259399 100644 --- a/doc/manual/write-PKGBUILD.en.md +++ b/doc/manual/write-PKGBUILD.en.md @@ -13,8 +13,9 @@ PKGBUILD includes variables and functions. - pkgrel: Mandatory. It indicates the release number of a software package. - arch: Mandatory. It indicates the architecture sequence of a software package. - depends: Optional. It indicates the name of the dependency package required for running a software test. -- makedepends: Optional. It indicates the list of files required for building a software package. -- source: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. +- makedepends: Optional. It indicates the name of the dependency package required for building a software package. +- source: Optional. It indicates the list of files required for building a software package. +- md5sums: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. ## Defining a Function diff --git a/doc/manual/write-PKGBUILD.zh.md b/doc/manual/write-PKGBUILD.zh.md index 414bfdc..ff53f0f 100644 --- a/doc/manual/write-PKGBUILD.zh.md +++ b/doc/manual/write-PKGBUILD.zh.md @@ -13,8 +13,9 @@ PKGBUILD�� - pkgrel: �� - arch: �� - depends: �� -- makedepends: �� -- source: ��MD5�� +- makedepends: �� +- source: �� +- md5sums: ��MD5�� ## �� -- 2.23.0

1 0

[PATCH lkp-tests] distro/archlinux: fix pacman search range
by Wang Yong 03 Feb '21

03 Feb '21

[before] $ pacman -Qqs gzip gzip zlib [after] $ pacman -Qq gzip gzip we need list local package accurately -s, --search <regexp> Search each locally-installed package for names or descriptions that match regexp. When including multiple search terms, only packages with descriptions matching ALL of those terms are returned. ref: https://archlinux.org/pacman/pacman.8.html Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/archlinux b/distro/archlinux index 3942080a6..d7b3d8d78 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -88,7 +88,7 @@ distro_install_depends() for pkg in $packages do - pacman -Qqs --noconfirm $pkg || pacman -Sy --noconfirm $pkg + pacman -Qq --noconfirm $pkg || pacman -Sy --noconfirm $pkg done done } -- 2.23.0

1 0

[PATCH v5 compass-ci 3/3] container/defconfig: load config yaml
by Liu Yinsi 03 Feb '21

03 Feb '21

Load "/etc/compass-ci/register/*.yaml", get register account info from cci defaults config. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index fd13312..f9b0824 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -9,6 +9,7 @@ def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', '/etc/compass-ci/accounts/*.yaml', + '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/3] container/auto-submit: bind mount volume
by Liu Yinsi 03 Feb '21

03 Feb '21

mount /etc/compass-ci/register, to get auto-submit account info(my_name, my_email) from config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..79463f6 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v5 compass-ci 1/3] sbin/auto-submit: config account yaml
by Liu Yinsi 03 Feb '21

03 Feb '21

[how] before: define my_name, my_email, lab, then store into ES and config yaml after: find my_name, my_email from /etc/compass-ci/register/*.yaml, then config yaml [why] store account info into ES, this part already done by container service register-accounts, so when start auto-submit service, no need to store data, only config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..c59deb7 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -6,7 +6,9 @@ require 'bunny' require 'json' require 'yaml' +require 'set' require_relative '../lib/build_my_info_client' +require "#{ENV['CCI_SRC']}/container/defconfig" # receive message and auto submit job class AutoSubmit @@ -86,15 +88,23 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def my_email + names = Set.new %w[auto-submit] + + defaults = relevant_defaults(names) + defaults['auto-submit']['my_email'] +end + +def config_yaml + build_my_info = BuildMyInfo.new(my_email) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] sparrow/1-storage: optimise format
by Liu Yinsi 03 Feb '21

03 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 47 ++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 0e8d722..0713d7e 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -1,26 +1,37 @@ #! /bin/bash -chown lkp:lkp /srv/result -chown lkp:lkp /srv/initrd -chown -R lkp:lkp /srv/es -chown -R mailer:team /srv/cci/Maildir -chown lkp:lkp /srv/cache/netdata_cache -chown lkp:lkp /srv/cache/netdata_lib -chown lkp:lkp /srv/cci/serial/fluentd-pos -chown lkp:lkp /srv/cci/libvirt-xml +chmod_dir=( + /srv/result + /srv/es/logging-es + /srv/es + /srv/cache/netdata_cache + /srv/cache/netdata_lib + /srv/cci/serial/logs + /srv/cci/libvirt-xml + /srv/cci/serial/fluentd-pos + /srv/cci/Maildir +) + +chmod 775 "${chmod_dir[@]}" + +chown_dir=( + /srv/result + /srv/initrd + /srv/es + /srv/cache/netdata_cache + /srv/cache/netdata_lib + /srv/cci/serial/fluentd-pos + /srv/cci/libvirt-xml +) + +chown lkp:lkp "${chown_dir[@]}" + chown lkp:committer /srv/git -chgrp team /srv/cci/serial/logs -chmod 775 /srv/result -chmod 775 /srv/es/logging-es -chmod 775 /srv/es -chmod 775 /srv/cache/netdata_cache -chmod 775 /srv/cache/netdata_lib -chmod 775 /srv/cci/serial/logs -chmod 775 /srv/cci/libvirt-xml -chmod 775 /srv/cci/serial/fluentd-pos -chmod -R 775 /srv/cci/Maildir +chown mailer:team /srv/cci/Maildir +chgrp team /srv/cci/serial/logs chgrp team /srv/dc + chmod g+ws /srv/dc find /tftpboot -type d | xargs chmod 775 -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sparrow/1-storage: add work dir and set permission
by Liu Yinsi 03 Feb '21

03 Feb '21

stores xml files required by libvirt Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index d1367cf..0e8d722 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -7,6 +7,7 @@ chown -R mailer:team /srv/cci/Maildir chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:lkp /srv/cci/libvirt-xml chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result @@ -15,6 +16,7 @@ chmod 775 /srv/es chmod 775 /srv/cache/netdata_cache chmod 775 /srv/cache/netdata_lib chmod 775 /srv/cci/serial/logs +chmod 775 /srv/cci/libvirt-xml chmod 775 /srv/cci/serial/fluentd-pos chmod -R 775 /srv/cci/Maildir diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8fd3652 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -17,6 +17,7 @@ dirs=( /srv/cci/scheduler /srv/cci/serial/logs /srv/cci/serial/fluentd-pos + /srv/cci/libvirt-xml /tftpboot /c /etc/qemu -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: optimize load config
by Wei Jihui 03 Feb '21

03 Feb '21

[why] current load config order can not load physical machine [how] optimize load config for: 1. load /etc/compass-ci/defaults/*.yaml load ~/.config/compass-ci/defaults/*.yaml --> get lab 2. load ~/.config/compass-ci/include/lab/$lab.yaml -->get my_token 3. now user get testbox, load load hosts config 4. load other file: like lkp-tests/include/* Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 477a8a12..9174cadc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -367,26 +367,15 @@ class Job return @job.include?('tbox_group') end - def get_lab_hosts_file - lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" - - if File.file?(lab_hosts_file) - lab_hosts_file - else - nil - end - end - def get_hosts_file - lab_hosts_file = get_lab_hosts_file - return lab_hosts_file if lab_hosts_file + testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" + return testbox_file if File.file?(testbox_file) - hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] - hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" - return hosts_file if File.file?(hosts_file) + tbox_group = @job['tbox_group'].split(/\.|--/)[0] + tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" + return tbox_group_file if File.file?(tbox_group_file) - raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" + raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end def include_files @@ -438,6 +427,9 @@ class Job lab_yaml = File.join(self_config_path, 'include/lab', "#{@defaults['lab']}.yaml") load_one_defaults(lab_yaml, @job) + + merge_defaults + load_hosts_config end def load_defaults(first_time = true) @@ -749,7 +741,6 @@ class Job each_job_init job = deepcopy(@job) load_defaults - load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| @@ -757,7 +748,6 @@ class Job @job = deepcopy(job) @job.merge!(hash) load_defaults - load_hosts_config each_job_init each_job(&block) end -- 2.23.0

1 0

[PATCH v3 compass-ci] delimiter/utils.rb: fix get the wrong job id when submit job failed
by Cao Xueliang 03 Feb '21

03 Feb '21

If we submit job failed, we get the response "xxx job id=0, error: xxx". We should match the response at first and return nil, otherwise we will get the wrong job id. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41b65de..3120ca6 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -69,6 +69,7 @@ module Utils save_job_to_yaml(job, PROCESS_JOB_YAML) response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" + return nil if response =~ /job id=0/ return $1 if response =~ /job id=(.*)/ return nil -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/1-storage: add work dir and set permission
by Liu Yinsi 03 Feb '21

03 Feb '21

stores xml files required by libvirt Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index d1367cf..0e8d722 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -7,6 +7,7 @@ chown -R mailer:team /srv/cci/Maildir chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:lkp /srv/cci/libvirt-xml chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result @@ -15,6 +16,7 @@ chmod 775 /srv/es chmod 775 /srv/cache/netdata_cache chmod 775 /srv/cache/netdata_lib chmod 775 /srv/cci/serial/logs +chmod 775 /srv/cci/libvirt-xml chmod 775 /srv/cci/serial/fluentd-pos chmod -R 775 /srv/cci/Maildir diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8fd3652 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -17,6 +17,7 @@ dirs=( /srv/cci/scheduler /srv/cci/serial/logs /srv/cci/serial/fluentd-pos + /srv/cci/libvirt-xml /tftpboot /c /etc/qemu -- 2.23.0

2 2

[PATCH v2 lkp-tests] installer/oracle: add shebang
by Wang Yong 03 Feb '21

03 Feb '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/installer/oracle | 2 ++ 1 file changed, 2 insertions(+) diff --git a/distro/installer/oracle b/distro/installer/oracle index b09abc945..11df716d6 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,3 +1,5 @@ +#!/bin/sh + if ! rpm -q epel-release >/dev/null; then yum install -y epel-release yum makecache fast -- 2.23.0

1 0

[PATCH lkp-tests] installer/oracle: add shebang
by Wang Yong 03 Feb '21

03 Feb '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/installer/oracle | 2 ++ 1 file changed, 2 insertions(+) diff --git a/distro/installer/oracle b/distro/installer/oracle index b09abc945..0831a5b6c 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,3 +1,5 @@ +#!/bin/bash + if ! rpm -q epel-release >/dev/null; then yum install -y epel-release yum makecache fast -- 2.23.0

2 2

[PATCH compass-ci] dnsmasq: open the dns function
by Yu Chuan 03 Feb '21

03 Feb '21

[Why] Generally, dnsmasq is used for a internal network lab, and as the number of machines in the lab increases, we need an internal dns server to perform dns relay and forward function. [How] 1. open dns function: - set 'port=53'; 2. set the upstream dns server of dnsmasq: - add param: 'resolv-conf'; - add resolv.conf and mount it to container. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dnsmasq/conf/etc/resolv.conf | 1 + container/dnsmasq/dnsmasq.d/dnsmasq.conf | 7 ++++++- container/dnsmasq/start | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 container/dnsmasq/conf/etc/resolv.conf diff --git a/container/dnsmasq/conf/etc/resolv.conf b/container/dnsmasq/conf/etc/resolv.conf new file mode 100644 index 000000000000..bad8aa25de25 --- /dev/null +++ b/container/dnsmasq/conf/etc/resolv.conf @@ -0,0 +1 @@ +nameserver 114.114.114.114 diff --git a/container/dnsmasq/dnsmasq.d/dnsmasq.conf b/container/dnsmasq/dnsmasq.d/dnsmasq.conf index 6141faf0436b..877216d8abdb 100644 --- a/container/dnsmasq/dnsmasq.d/dnsmasq.conf +++ b/container/dnsmasq/dnsmasq.d/dnsmasq.conf @@ -1,4 +1,9 @@ -port=0 +# open dns function +port=53 + +# setup the upstream dns server of dnsmasq +resolv-file=/etc/resolv.conf + interface=br0 enable-tftp diff --git a/container/dnsmasq/start b/container/dnsmasq/start index ff4247046eb1..a75ff209e64f 100755 --- a/container/dnsmasq/start +++ b/container/dnsmasq/start @@ -27,6 +27,7 @@ cmd=( --publish 67:67/udp --publish 69:69/udp -v ${WORKSPACE}/dnsmasq.d:/etc/dnsmasq.d + -v ${WORKSPACE}/conf/etc/resolv.conf:/etc/resolv.conf:ro # specify the upstream dns server of dnsmasq -v ${DNSMASQ_CONF_DIR}/lease_dir:/var/lib/misc # persist the dhcp lease file -v /etc/localtime:/etc/localtime:ro -v /tftpboot:/tftpboot:ro -- 2.23.0

1 0

[PATCH v2 compass-ci] delimiter/utils.rb: deal submit job failed
by Cao Xueliang 03 Feb '21

03 Feb '21

If we submit job failed, we get the response "xxx job id=0, error: xxx", we should match the response at first and return nil. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41b65de..3120ca6 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -69,6 +69,7 @@ module Utils save_job_to_yaml(job, PROCESS_JOB_YAML) response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" + return nil if response =~ /job id=0/ return $1 if response =~ /job id=(.*)/ return nil -- 2.23.0

1 0

[PATCH v3 compass-ci] container/openresty-proxy-cache: fix build error
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute openresty-proxy-cache/build, if openresty already exists [error] fatal: destination path 'openresty' already exists and is not an empty directory. [how] delete openresty before git clone. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/openresty-proxy-cache/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 0846197..507c4fe 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[ -d "openresty" ] && rm -rf openresty + git clone https://gitee.com/cuiyili/openresty.git || exit 1 docker build -t openresty:proxy_cache . -f openresty/Dockerfile -- 2.23.0

1 0

[PATCH v5 compass-ci 4/4] container/submit: README file
by Luan Shengde 03 Feb '21

03 Feb '21

help file for the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..6772961 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,26 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit most of the linux OSes. +# Avoid much repetitive testing work to adapt various OSes. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your local host. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v5 compass-ci 3/4] container/submit: executable file submit
by Luan Shengde 03 Feb '21

03 Feb '21

executable file for container: submit file 'submit' is used as the command installed in the container. it can be run as the real command submit itself in the container. when you run it, it will calls the command 'submit' in the container. add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..4e7e24f --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This is a single-use container. +# The container will run if you run the 'submit' to submit jobs. +# After the command execute over, the container will been removed itself. +# Do not need to delete manually for it has been added option '--rm'. +date_suffix=$(date +%m%d-%H%M%S) +cmd=( + docker run + --rm + --name=submit-$USER-$data_suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v5 compass-ci 2/4] container/submit: build file
by Luan Shengde 03 Feb '21

03 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH compass-ci] sec/lib/web-backend.rb: fix runtime warning
by Lu Weitao 03 Feb '21

03 Feb '21

[Why] /home/lwt/compass-ci/src/lib/web_backend.rb:9: warning: already initialized constant CCI_SRC ./web-backend:8: warning: previous definition of CCI_SRC was here Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..7cd34f8 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -6,7 +6,7 @@ require 'json' require 'yaml' require 'set' -CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' +CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils.rb: deal submit job failed
by Cao Xueliang 03 Feb '21

03 Feb '21

If we submit job failed, we get the response "xxx job id=0, error: xxx", we should match the response at first and return nil Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41b65de..3120ca6 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -69,6 +69,7 @@ module Utils save_job_to_yaml(job, PROCESS_JOB_YAML) response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" + return nil if response =~ /job id=0/ return $1 if response =~ /job id=(.*)/ return nil -- 2.23.0

1 0

[PATCH compass-ci] providers/lib: introduce domain.rb action.rb
by Xiao Shenwei 03 Feb '21

03 Feb '21

function: based on response of boot.libvirt generate a domain.xml refer-to: commit 457644c0486e0786ed8b563c6e4928445eea0207 if you want to test with your domain.xml, define vt field in job.yaml vt: domain: xxx/xxx.xml if not will use default options to generate a domain.xml TODO: support combined templates --- providers/lib/action.rb | 75 +++++++++++++++++++++++++++++++++++++++++ providers/lib/domain.rb | 75 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 providers/lib/action.rb create mode 100644 providers/lib/domain.rb diff --git a/providers/lib/action.rb b/providers/lib/action.rb new file mode 100644 index 0000000..526d84d --- /dev/null +++ b/providers/lib/action.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# expand each element of the domain.xml +module Action + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + def expand_erb(file) + @context.expand_erb(File.read(file)) + end + + def domain + @doc = Nokogiri::XML(expand_erb(@domain_option)) + end + + def name + @doc.xpath('//domain/name').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/name.xml") + end + + def os + @doc.xpath('//domain/os').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/os.xml") + end + + def emulator + @doc.xpath('//domain/devices/emulator').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/emulator.xml") + end + + def cpu + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/cpu.xml") + end + + def memory + @doc.xpath('//domain/memory').remove + @doc.xpath('//domain/currentMemory').remove + @doc.xpath('//domain/maxMemory').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/memory.xml") + end + + def devices + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/devices.xml") + end + + def disk + @context.info['disk'] ||= nil + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/disk.xml") + end + + def serial + @doc.xpath('//domain/devices/serial').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/serial.xml") + end + + def interface + @doc.xpath('//domain/devices/interface').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/interface.xml") + end + + def active + @doc.xpath('//domain/on_poweroff').remove + @doc.xpath('//domain/on_reboot').remove + @doc.xpath('//domain/on_crash').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/active.xml") + end + + def clock + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/clock.xml") + end + + def seclabel + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/seclabel.xml") + end +end diff --git a/providers/lib/domain.rb b/providers/lib/domain.rb new file mode 100644 index 0000000..8df9c06 --- /dev/null +++ b/providers/lib/domain.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'set' +require 'nokogiri' +require 'yaml' +require_relative 'action' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# generate a domain.xml +class Domain + include Action + + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + OPTION_FILE = "#{TEMPLATE_DIR}/options.yaml" + def initialize(context, logger) + @doc = nil + @context = context + @logger = logger + @options = Hashugar.new(YAML.safe_load(File.read(OPTION_FILE))) + end + + def generate + domain_option + if user_domain? + default_option + return + end + replaceable_option + default_option + end + + def save_to(filename) + File.open(filename, 'w') do |f| + f.puts @doc.to_xml + end + File.realpath(filename) + end + + private + + def user_domain? + @context.info['vt'].key?('domain') + end + + def load_xml_file(filepath) + host = @context.info['SRV_HTTP_HOST'] + port = @context.info['SRV_HTTP_PORT'] + system "wget --timestamping --progress=bar:force http://#{host}:#{port}/cci/libvirt-xml/#{filepath}" + %x(basename #{filepath}).chomp + end + + def domain_option + @domain_option = "#{TEMPLATE_DIR}/#{(a)options.domain}.xml" + if user_domain? + @domain_option = load_xml_file((a)context.info['vt']['domain']) + end + @logger.info("Domain option: #{@domain_option}") + domain + end + + def default_option + @options.default.each do |one| + instance_eval one + end + end + + def replaceable_option + @options.replaceable.each do |one| + instance_eval one + end + end +end -- 2.23.0

1 0

[PATCH lkp-tests] distro/debian: force install without upgrade
by Wang Yong 03 Feb '21

03 Feb '21

[Why] when install exist package, it will be upgrade and may throw many dpkg warnings like: files list file for package 'libgme0:arm64' missing; assuming package has no files currently installed now force install package without upgrade it by add --no-upgrade --no-upgrade Do not upgrade packages. When used in conjunction with install, no-upgrade will prevent packages listed from being upgraded if they are already installed. ref: https://linux.die.net/man/8/apt-get Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 8821e3ddd..9b6636f48 100755 --- a/distro/debian +++ b/distro/debian @@ -193,7 +193,7 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y $package + install -y --no-upgrade $package done done } -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker: split function to build and run container
by Liu Yinsi 03 Feb '21

03 Feb '21

[how] before: build_depends -> do_one: build and run container's depends and itself in parallel after: build_depends -> do_one_build: build container's depends and itself in parallel start_depends -> do_one_run: run container's depends and itself not in parallel [why] build_depends just build container, start_depends just run container, it will make logic more clear. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 53 ++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c676e6f..d4e9419 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -17,10 +17,10 @@ build_depends() done wait - do_one $container + do_one_build $container } -do_one() +do_one_build() { local container=$1 local container_name=$(basename $container) @@ -29,30 +29,53 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - log_info "start build $container." + log_info "start build $container." - [ -x build ] && ./build - [ -x install ] && ./install + [ -x build ] && ./build + [ -x install ] && ./install - log_info "finish build $container." - fi - [ -x first-run ] && ./first-run - [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] && ./run + log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } +start_depends() +{ + local container=$1 + + for dep in $(cat $container/start-depends 2> /dev/null) + do + start_depends $CONTAINER_PATH/$dep + done + + do_one_run $container +} + +do_one_run() +{ + local container=$1 + local container_name=$(basename $container) + + cd "$container" + [ -x first-run ] && ./first-run + [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run +} + tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir & + container_name=$(basename $dir) + [ "$container_name" == 'ssh-r' ] && continue + + docker images | grep -wq "$container_name" + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then + build_depends $dir & + wait + fi + start_depends $dir done -wait rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] container/delimiter: fix failed to build docker image
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute delimiter/build: `push_image delimiter:latest` [error] The push refers to repository [$DOCKER_REGISTRY_HOST:5001/delimiter] Get http://$DOCKER_REGISTRY_PORT:5001/v2/: dial tcp $DOCKER_REGISTRY_HOST:5001:connect: connection refused [why] container registry service port is 5001, connect failed 5001 because registry not ready, so add build-depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/delimiter/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/delimiter/build-depends diff --git a/container/delimiter/build-depends b/container/delimiter/build-depends new file mode 100755 index 0000000..14bc599 --- /dev/null +++ b/container/delimiter/build-depends @@ -0,0 +1 @@ +registry -- 2.23.0

3 2

[PATCH v2 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 03 Feb '21

03 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 62 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..141335c3ee4b --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,62 @@ +# Summary +========= + +`os_mount` defines the mount method of tbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + optional kernel cmdline params: # can be added in kernel_append_root field of job.yaml + - keep_last_root: keep the original data of disk what last job generated, don't wipe it + +2. scheduler return the ipxe_str to tbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. boot + - check if `/dev/mapper/os-openeuler_aarch64_20.03_{timestamp}` exist # according the kernel cmdline + - if exist: + - if `keep_last_root` not in kernel cmdline params: + - re-create(lvremove and lvcreate) rw logical volume /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if `keep_last_root` in kernel cmdline params: + - if /dev/mapper/os-openeuler_aarch64_20.03 exists: + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if /dev/mapper/os-openeuler_aarch64_20.03 not exists: + - generate /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03-{timestamp} + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if not exist: + ``` + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + create /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} according rootfs_disk. # logical volume size: 10G + mkdir /mnt1, mount /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} /mnt1 + rsync -az /mnt/. /mnt1/ + umount /mnt /mnt1 + lvchange -p r /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + creat rw logical volume /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + boot from /dev/mapper/os-openeuler_aarch64_20.03 + ``` + +4. execute the job -- 2.23.0

2 3

2024

2023

2022

2021

2020

Compass-ci