- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] container/openresty-proxy-cache: fix build error
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute openresty-proxy-cache/build, if openresty already exists [error] fatal: destination path 'openresty' already exists and is not an empty directory. [how] delete openresty before git clone. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/openresty-proxy-cache/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 0846197..507c4fe 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[ -d "openresty" ] && rm -rf openresty + git clone https://gitee.com/cuiyili/openresty.git || exit 1 docker build -t openresty:proxy_cache . -f openresty/Dockerfile -- 2.23.0

1 0

[PATCH v5 compass-ci 4/4] container/submit: README file
by Luan Shengde 03 Feb '21

03 Feb '21

help file for the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..6772961 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,26 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit most of the linux OSes. +# Avoid much repetitive testing work to adapt various OSes. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your local host. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v5 compass-ci 3/4] container/submit: executable file submit
by Luan Shengde 03 Feb '21

03 Feb '21

executable file for container: submit file 'submit' is used as the command installed in the container. it can be run as the real command submit itself in the container. when you run it, it will calls the command 'submit' in the container. add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..4e7e24f --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This is a single-use container. +# The container will run if you run the 'submit' to submit jobs. +# After the command execute over, the container will been removed itself. +# Do not need to delete manually for it has been added option '--rm'. +date_suffix=$(date +%m%d-%H%M%S) +cmd=( + docker run + --rm + --name=submit-$USER-$data_suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v5 compass-ci 2/4] container/submit: build file
by Luan Shengde 03 Feb '21

03 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH compass-ci] sec/lib/web-backend.rb: fix runtime warning
by Lu Weitao 03 Feb '21

03 Feb '21

[Why] /home/lwt/compass-ci/src/lib/web_backend.rb:9: warning: already initialized constant CCI_SRC ./web-backend:8: warning: previous definition of CCI_SRC was here Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..7cd34f8 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -6,7 +6,7 @@ require 'json' require 'yaml' require 'set' -CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' +CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils.rb: deal submit job failed
by Cao Xueliang 03 Feb '21

03 Feb '21

If we submit job failed, we get the response "xxx job id=0, error: xxx", we should match the response at first and return nil Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41b65de..3120ca6 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -69,6 +69,7 @@ module Utils save_job_to_yaml(job, PROCESS_JOB_YAML) response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" + return nil if response =~ /job id=0/ return $1 if response =~ /job id=(.*)/ return nil -- 2.23.0

1 0

[PATCH compass-ci] providers/lib: introduce domain.rb action.rb
by Xiao Shenwei 03 Feb '21

03 Feb '21

function: based on response of boot.libvirt generate a domain.xml refer-to: commit 457644c0486e0786ed8b563c6e4928445eea0207 if you want to test with your domain.xml, define vt field in job.yaml vt: domain: xxx/xxx.xml if not will use default options to generate a domain.xml TODO: support combined templates --- providers/lib/action.rb | 75 +++++++++++++++++++++++++++++++++++++++++ providers/lib/domain.rb | 75 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 providers/lib/action.rb create mode 100644 providers/lib/domain.rb diff --git a/providers/lib/action.rb b/providers/lib/action.rb new file mode 100644 index 0000000..526d84d --- /dev/null +++ b/providers/lib/action.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# expand each element of the domain.xml +module Action + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + def expand_erb(file) + @context.expand_erb(File.read(file)) + end + + def domain + @doc = Nokogiri::XML(expand_erb(@domain_option)) + end + + def name + @doc.xpath('//domain/name').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/name.xml") + end + + def os + @doc.xpath('//domain/os').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/os.xml") + end + + def emulator + @doc.xpath('//domain/devices/emulator').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/emulator.xml") + end + + def cpu + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/cpu.xml") + end + + def memory + @doc.xpath('//domain/memory').remove + @doc.xpath('//domain/currentMemory').remove + @doc.xpath('//domain/maxMemory').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/memory.xml") + end + + def devices + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/devices.xml") + end + + def disk + @context.info['disk'] ||= nil + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/disk.xml") + end + + def serial + @doc.xpath('//domain/devices/serial').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/serial.xml") + end + + def interface + @doc.xpath('//domain/devices/interface').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/interface.xml") + end + + def active + @doc.xpath('//domain/on_poweroff').remove + @doc.xpath('//domain/on_reboot').remove + @doc.xpath('//domain/on_crash').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/active.xml") + end + + def clock + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/clock.xml") + end + + def seclabel + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/seclabel.xml") + end +end diff --git a/providers/lib/domain.rb b/providers/lib/domain.rb new file mode 100644 index 0000000..8df9c06 --- /dev/null +++ b/providers/lib/domain.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'set' +require 'nokogiri' +require 'yaml' +require_relative 'action' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# generate a domain.xml +class Domain + include Action + + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + OPTION_FILE = "#{TEMPLATE_DIR}/options.yaml" + def initialize(context, logger) + @doc = nil + @context = context + @logger = logger + @options = Hashugar.new(YAML.safe_load(File.read(OPTION_FILE))) + end + + def generate + domain_option + if user_domain? + default_option + return + end + replaceable_option + default_option + end + + def save_to(filename) + File.open(filename, 'w') do |f| + f.puts @doc.to_xml + end + File.realpath(filename) + end + + private + + def user_domain? + @context.info['vt'].key?('domain') + end + + def load_xml_file(filepath) + host = @context.info['SRV_HTTP_HOST'] + port = @context.info['SRV_HTTP_PORT'] + system "wget --timestamping --progress=bar:force http://#{host}:#{port}/cci/libvirt-xml/#{filepath}" + %x(basename #{filepath}).chomp + end + + def domain_option + @domain_option = "#{TEMPLATE_DIR}/#{(a)options.domain}.xml" + if user_domain? + @domain_option = load_xml_file((a)context.info['vt']['domain']) + end + @logger.info("Domain option: #{@domain_option}") + domain + end + + def default_option + @options.default.each do |one| + instance_eval one + end + end + + def replaceable_option + @options.replaceable.each do |one| + instance_eval one + end + end +end -- 2.23.0

1 0

[PATCH lkp-tests] distro/debian: force install without upgrade
by Wang Yong 03 Feb '21

03 Feb '21

[Why] when install exist package, it will be upgrade and may throw many dpkg warnings like: files list file for package 'libgme0:arm64' missing; assuming package has no files currently installed now force install package without upgrade it by add --no-upgrade --no-upgrade Do not upgrade packages. When used in conjunction with install, no-upgrade will prevent packages listed from being upgraded if they are already installed. ref: https://linux.die.net/man/8/apt-get Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 8821e3ddd..9b6636f48 100755 --- a/distro/debian +++ b/distro/debian @@ -193,7 +193,7 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y $package + install -y --no-upgrade $package done done } -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker: split function to build and run container
by Liu Yinsi 03 Feb '21

03 Feb '21

[how] before: build_depends -> do_one: build and run container's depends and itself in parallel after: build_depends -> do_one_build: build container's depends and itself in parallel start_depends -> do_one_run: run container's depends and itself not in parallel [why] build_depends just build container, start_depends just run container, it will make logic more clear. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 53 ++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c676e6f..d4e9419 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -17,10 +17,10 @@ build_depends() done wait - do_one $container + do_one_build $container } -do_one() +do_one_build() { local container=$1 local container_name=$(basename $container) @@ -29,30 +29,53 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - log_info "start build $container." + log_info "start build $container." - [ -x build ] && ./build - [ -x install ] && ./install + [ -x build ] && ./build + [ -x install ] && ./install - log_info "finish build $container." - fi - [ -x first-run ] && ./first-run - [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] && ./run + log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } +start_depends() +{ + local container=$1 + + for dep in $(cat $container/start-depends 2> /dev/null) + do + start_depends $CONTAINER_PATH/$dep + done + + do_one_run $container +} + +do_one_run() +{ + local container=$1 + local container_name=$(basename $container) + + cd "$container" + [ -x first-run ] && ./first-run + [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run +} + tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir & + container_name=$(basename $dir) + [ "$container_name" == 'ssh-r' ] && continue + + docker images | grep -wq "$container_name" + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then + build_depends $dir & + wait + fi + start_depends $dir done -wait rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] container/delimiter: fix failed to build docker image
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute delimiter/build: `push_image delimiter:latest` [error] The push refers to repository [$DOCKER_REGISTRY_HOST:5001/delimiter] Get http://$DOCKER_REGISTRY_PORT:5001/v2/: dial tcp $DOCKER_REGISTRY_HOST:5001:connect: connection refused [why] container registry service port is 5001, connect failed 5001 because registry not ready, so add build-depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/delimiter/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/delimiter/build-depends diff --git a/container/delimiter/build-depends b/container/delimiter/build-depends new file mode 100755 index 0000000..14bc599 --- /dev/null +++ b/container/delimiter/build-depends @@ -0,0 +1 @@ +registry -- 2.23.0

3 2