- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/4] container/submit: build file
by Luan Shengde 03 Feb '21

03 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH compass-ci] sec/lib/web-backend.rb: fix runtime warning
by Lu Weitao 03 Feb '21

03 Feb '21

[Why] /home/lwt/compass-ci/src/lib/web_backend.rb:9: warning: already initialized constant CCI_SRC ./web-backend:8: warning: previous definition of CCI_SRC was here Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..7cd34f8 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -6,7 +6,7 @@ require 'json' require 'yaml' require 'set' -CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' +CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils.rb: deal submit job failed
by Cao Xueliang 03 Feb '21

03 Feb '21

If we submit job failed, we get the response "xxx job id=0, error: xxx", we should match the response at first and return nil Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41b65de..3120ca6 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -69,6 +69,7 @@ module Utils save_job_to_yaml(job, PROCESS_JOB_YAML) response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" + return nil if response =~ /job id=0/ return $1 if response =~ /job id=(.*)/ return nil -- 2.23.0

1 0

[PATCH compass-ci] providers/lib: introduce domain.rb action.rb
by Xiao Shenwei 03 Feb '21

03 Feb '21

function: based on response of boot.libvirt generate a domain.xml refer-to: commit 457644c0486e0786ed8b563c6e4928445eea0207 if you want to test with your domain.xml, define vt field in job.yaml vt: domain: xxx/xxx.xml if not will use default options to generate a domain.xml TODO: support combined templates --- providers/lib/action.rb | 75 +++++++++++++++++++++++++++++++++++++++++ providers/lib/domain.rb | 75 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 providers/lib/action.rb create mode 100644 providers/lib/domain.rb diff --git a/providers/lib/action.rb b/providers/lib/action.rb new file mode 100644 index 0000000..526d84d --- /dev/null +++ b/providers/lib/action.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# expand each element of the domain.xml +module Action + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + def expand_erb(file) + @context.expand_erb(File.read(file)) + end + + def domain + @doc = Nokogiri::XML(expand_erb(@domain_option)) + end + + def name + @doc.xpath('//domain/name').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/name.xml") + end + + def os + @doc.xpath('//domain/os').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/os.xml") + end + + def emulator + @doc.xpath('//domain/devices/emulator').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/emulator.xml") + end + + def cpu + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/cpu.xml") + end + + def memory + @doc.xpath('//domain/memory').remove + @doc.xpath('//domain/currentMemory').remove + @doc.xpath('//domain/maxMemory').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/memory.xml") + end + + def devices + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/devices.xml") + end + + def disk + @context.info['disk'] ||= nil + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/disk.xml") + end + + def serial + @doc.xpath('//domain/devices/serial').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/serial.xml") + end + + def interface + @doc.xpath('//domain/devices/interface').remove + @doc.xpath('//domain/devices')[0].add_child expand_erb("#{TEMPLATE_DIR}/interface.xml") + end + + def active + @doc.xpath('//domain/on_poweroff').remove + @doc.xpath('//domain/on_reboot').remove + @doc.xpath('//domain/on_crash').remove + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/active.xml") + end + + def clock + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/clock.xml") + end + + def seclabel + @doc.root.add_child expand_erb("#{TEMPLATE_DIR}/seclabel.xml") + end +end diff --git a/providers/lib/domain.rb b/providers/lib/domain.rb new file mode 100644 index 0000000..8df9c06 --- /dev/null +++ b/providers/lib/domain.rb @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'set' +require 'nokogiri' +require 'yaml' +require_relative 'action' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# generate a domain.xml +class Domain + include Action + + TEMPLATE_DIR = "#{ENV['CCI_SRC']}/providers/libvirt/templates" + OPTION_FILE = "#{TEMPLATE_DIR}/options.yaml" + def initialize(context, logger) + @doc = nil + @context = context + @logger = logger + @options = Hashugar.new(YAML.safe_load(File.read(OPTION_FILE))) + end + + def generate + domain_option + if user_domain? + default_option + return + end + replaceable_option + default_option + end + + def save_to(filename) + File.open(filename, 'w') do |f| + f.puts @doc.to_xml + end + File.realpath(filename) + end + + private + + def user_domain? + @context.info['vt'].key?('domain') + end + + def load_xml_file(filepath) + host = @context.info['SRV_HTTP_HOST'] + port = @context.info['SRV_HTTP_PORT'] + system "wget --timestamping --progress=bar:force http://#{host}:#{port}/cci/libvirt-xml/#{filepath}" + %x(basename #{filepath}).chomp + end + + def domain_option + @domain_option = "#{TEMPLATE_DIR}/#{(a)options.domain}.xml" + if user_domain? + @domain_option = load_xml_file((a)context.info['vt']['domain']) + end + @logger.info("Domain option: #{@domain_option}") + domain + end + + def default_option + @options.default.each do |one| + instance_eval one + end + end + + def replaceable_option + @options.replaceable.each do |one| + instance_eval one + end + end +end -- 2.23.0

1 0

[PATCH lkp-tests] distro/debian: force install without upgrade
by Wang Yong 03 Feb '21

03 Feb '21

[Why] when install exist package, it will be upgrade and may throw many dpkg warnings like: files list file for package 'libgme0:arm64' missing; assuming package has no files currently installed now force install package without upgrade it by add --no-upgrade --no-upgrade Do not upgrade packages. When used in conjunction with install, no-upgrade will prevent packages listed from being upgraded if they are already installed. ref: https://linux.die.net/man/8/apt-get Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 8821e3ddd..9b6636f48 100755 --- a/distro/debian +++ b/distro/debian @@ -193,7 +193,7 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y $package + install -y --no-upgrade $package done done } -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker: split function to build and run container
by Liu Yinsi 03 Feb '21

03 Feb '21

[how] before: build_depends -> do_one: build and run container's depends and itself in parallel after: build_depends -> do_one_build: build container's depends and itself in parallel start_depends -> do_one_run: run container's depends and itself not in parallel [why] build_depends just build container, start_depends just run container, it will make logic more clear. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 53 ++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c676e6f..d4e9419 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -17,10 +17,10 @@ build_depends() done wait - do_one $container + do_one_build $container } -do_one() +do_one_build() { local container=$1 local container_name=$(basename $container) @@ -29,30 +29,53 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - log_info "start build $container." + log_info "start build $container." - [ -x build ] && ./build - [ -x install ] && ./install + [ -x build ] && ./build + [ -x install ] && ./install - log_info "finish build $container." - fi - [ -x first-run ] && ./first-run - [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] && ./run + log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } +start_depends() +{ + local container=$1 + + for dep in $(cat $container/start-depends 2> /dev/null) + do + start_depends $CONTAINER_PATH/$dep + done + + do_one_run $container +} + +do_one_run() +{ + local container=$1 + local container_name=$(basename $container) + + cd "$container" + [ -x first-run ] && ./first-run + [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run +} + tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir & + container_name=$(basename $dir) + [ "$container_name" == 'ssh-r' ] && continue + + docker images | grep -wq "$container_name" + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then + build_depends $dir & + wait + fi + start_depends $dir done -wait rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] container/delimiter: fix failed to build docker image
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute delimiter/build: `push_image delimiter:latest` [error] The push refers to repository [$DOCKER_REGISTRY_HOST:5001/delimiter] Get http://$DOCKER_REGISTRY_PORT:5001/v2/: dial tcp $DOCKER_REGISTRY_HOST:5001:connect: connection refused [why] container registry service port is 5001, connect failed 5001 because registry not ready, so add build-depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/delimiter/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/delimiter/build-depends diff --git a/container/delimiter/build-depends b/container/delimiter/build-depends new file mode 100755 index 0000000..14bc599 --- /dev/null +++ b/container/delimiter/build-depends @@ -0,0 +1 @@ +registry -- 2.23.0

3 2

[PATCH v2 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 03 Feb '21

03 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 62 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..141335c3ee4b --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,62 @@ +# Summary +========= + +`os_mount` defines the mount method of tbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + optional kernel cmdline params: # can be added in kernel_append_root field of job.yaml + - keep_last_root: keep the original data of disk what last job generated, don't wipe it + +2. scheduler return the ipxe_str to tbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. boot + - check if `/dev/mapper/os-openeuler_aarch64_20.03_{timestamp}` exist # according the kernel cmdline + - if exist: + - if `keep_last_root` not in kernel cmdline params: + - re-create(lvremove and lvcreate) rw logical volume /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if `keep_last_root` in kernel cmdline params: + - if /dev/mapper/os-openeuler_aarch64_20.03 exists: + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if /dev/mapper/os-openeuler_aarch64_20.03 not exists: + - generate /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03-{timestamp} + - boot from /dev/mapper/os-openeuler_aarch64_20.03 + - if not exist: + ``` + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + create /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} according rootfs_disk. # logical volume size: 10G + mkdir /mnt1, mount /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} /mnt1 + rsync -az /mnt/. /mnt1/ + umount /mnt /mnt1 + lvchange -p r /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + creat rw logical volume /dev/mapper/os-openeuler_aarch64_20.03 from /dev/mapper/os-openeuler_aarch64_20.03_{timestamp} + boot from /dev/mapper/os-openeuler_aarch64_20.03 + ``` + +4. execute the job -- 2.23.0

2 3

[PATCH compass-ci] register-accounts: sync register-accounts
by Luan Shengde 03 Feb '21

03 Feb '21

[why] the required file lib/build_my_info_client has been updated. and the invoking way has some change. just do update according to the change. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/register-accounts/register-accounts | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/container/register-accounts/register-accounts b/container/register-accounts/register-accounts index ff69032..519f289 100755 --- a/container/register-accounts/register-accounts +++ b/container/register-accounts/register-accounts @@ -9,7 +9,12 @@ require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" content = YAML.load_file('/etc/compass-ci/register/register.yaml') content.each do |key, value| - build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) - build_my_info.store_account_info + my_info = {} + my_info['my_email'] = value['my_email'] + my_info['my_name'] = value['my_name'] + my_info['lab'] = ENV['lab'] + + build_my_info = BuildMyInfo.new(my_info['my_email']) + build_my_info.store_account_info(my_info) end sleep -- 2.23.0

2 1

[PATCH v2 compass-ci] container/openresty-proxy-cache: fix build error
by Liu Yinsi 03 Feb '21

03 Feb '21

when execute openresty-proxy-cache/build, if openresty already exists [error] fatal: destination path 'openresty' already exists and is not an empty directory. [how] delete openresty before git clone. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/openresty-proxy-cache/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 0846197..2c982f7 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +rm -rf openresty + git clone https://gitee.com/cuiyili/openresty.git || exit 1 docker build -t openresty:proxy_cache . -f openresty/Dockerfile -- 2.23.0

1 0

[PATCH compass-ci] sparrow/1-storage: add work dir and set permission
by Liu Yinsi 03 Feb '21

03 Feb '21

add work dir and permission to use libvirt to boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index d1367cf..0e8d722 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -7,6 +7,7 @@ chown -R mailer:team /srv/cci/Maildir chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:lkp /srv/cci/libvirt-xml chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result @@ -15,6 +16,7 @@ chmod 775 /srv/es chmod 775 /srv/cache/netdata_cache chmod 775 /srv/cache/netdata_lib chmod 775 /srv/cci/serial/logs +chmod 775 /srv/cci/libvirt-xml chmod 775 /srv/cci/serial/fluentd-pos chmod -R 775 /srv/cci/Maildir diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8fd3652 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -17,6 +17,7 @@ dirs=( /srv/cci/scheduler /srv/cci/serial/logs /srv/cci/serial/fluentd-pos + /srv/cci/libvirt-xml /tftpboot /c /etc/qemu -- 2.23.0

2 2

Re: [PATCH compass-ci] container/openresty-proxy-cache: fix build error
by Liu Yinsi 03 Feb '21

03 Feb '21

>> git clone https://gitee.com/cuiyili/openresty.git || exit 1 >> docker build -t openresty:proxy_cache . -f openresty/Dockerfile >>+rm -rf openresty > >How about move the 'rm' step at the before of 'docker build'? docker build need openrestry/Dockerfile, this file come from git clone https://gitee.com/cuiyili/openresty.git, you mean move before of git clone? it looks not make any difference. Thanks, Yinsi > >-------- >Thanks >Yu Chuan > >>-- >>2.23.0 >>

2 2

[PATCH v5 compass-ci 3/3] mail-robot: delete useless file
by Luan Shengde 03 Feb '21

03 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 45 ------------------- 1 file changed, 45 deletions(-) delete mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb deleted file mode 100755 index af31e17..0000000 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -require 'json' -require 'mail' - -# used to check commit available for hub gitee.com -# it is called when: -# - ParseApplyAccountEmail execute commit_url_availability -# - hub is gitee.com -# gitee_commit_check -# clone the repo -# check commit available -class GiteeCommitUrlCheck - def initialize(my_info, url, base_url) - @my_info = my_info - @url = url - @base_url = base_url - end - - def gitee_commit_check - repo_url = [@base_url, 'git'].join('.') - repo_dir = repo_url.split('/')[-1] - commit_id = @url.split('/')[-1] - - Dir.chdir '/tmp' - %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) - - email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] - - FileUtils.rm_rf repo_dir - - gitee_commit_exist(email_index) - end - - def gitee_commit_exist(email_index) - return if email_index - - error_message = 'COMMIT_URL_NOT_AVAILABLE' - - raise error_message - end -end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/3] mail-robot: optimize assign-account-fail-email
by Luan Shengde 03 Feb '21

03 Feb '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..3597159 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Choose a commit that was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + There is no such commit ID in the repo. - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit ID correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH v5 compass-ci 1/3] mail-robot: fix bug for parsing email
by Luan Shengde 03 Feb '21

03 Feb '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 41 ++++++++++--------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..c7fe9ea 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,33 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + %x(/usr/bin/git -C /tmp clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(/usr/bin/git -C /tmp/#{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C /tmp/#{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf "/tmp/#{repo_dir}" end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id.chomp + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'].chomp - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker: delete duplicate functions code
by Liu Yinsi 03 Feb '21

03 Feb '21

git show d265997c: buildall: avoid repeated building containers in sparrow git show 67ad8b6e: sparrow/4-docker/buildall: prevent duplicate build [why] delete one of the preceding patches because the two patches have the same functions. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 -- 1 file changed, 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 727abb9..c838f86 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -25,7 +25,6 @@ do_one() local container=$1 local container_name=$(basename $container) lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock - echo "$started_containers" | grep -w -q "$container_name" && return 0 log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && @@ -42,7 +41,6 @@ do_one() [ "$container_name" == 'initrd-lkp' ] && ./run ) log_info "finish build $container." - started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

2 1

[PATCH compass-ci] sparrow/4-docker: move confusing log_info()
by Liu Yinsi 03 Feb '21

03 Feb '21

[how] before: whether build container or not, all will call log_info(): "start/finish build $container" after: only when actually build container, will call log_info(): "start/finish build $container" [why] if not build container, but echo "start build $container", it's very confusing, mistakenly think that repeatedly build container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c838f86..c676e6f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -26,21 +26,24 @@ do_one() local container_name=$(basename $container) lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock - log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then + + log_info "start build $container." + [ -x build ] && ./build [ -x install ] && ./install + + log_info "finish build $container." fi [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run ) - log_info "finish build $container." lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

2 1

[PATCH lkp-tests] iso2rootfs: add new config_rootfs field: enable_repos
by Yu Chuan 03 Feb '21

03 Feb '21

[Why] Not all sources of one repo file are enabled, sometimes we need to enable all sources to install the specified pkg. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 1 + tests/iso2rootfs | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 2a6455245584..55c4345fad94 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -32,6 +32,7 @@ iso2rootfs: # - dns: will config /etc/resolv.conf. # - no_selinux: will disable selinux in /etc/selinux/config. # - no_fstab: will comment all line in /etc/fstab. + # - enable_repos: will enable all repo file of result rootfs. config_rootfs: ## install pkgs for result rootfs diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 65a1935b7f49..06c8565110ab 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -393,11 +393,23 @@ install_pkgs_for_rootfs() fi } +enable_repos() +{ + echo "${config_rootfs}" | grep -qw 'enable_repos' || return 0 + + ls ${ROOTFS_DES_DIR}/etc/yum.repos.d/*.repo > /dev/null 2>&1 && { + sed -i 's/^enabled=0$/enabled=1/g' ${ROOTFS_DES_DIR}/etc/yum.repos.d/*.repo + } + + return 0 +} + config_rootfs() { config_dns_resolver disable_selinux disable_fstab + enable_repos install_pkgs_for_rootfs } -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] sbin/build-my-info: optimize
by Luan Shengde 03 Feb '21

03 Feb '21

optimize options: add option -t: generate new my_token add option -h: output help message add search my_info from ES and update my_info before store it. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..dbe093d 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -8,26 +8,48 @@ require_relative '../lib/build_my_info_client' require 'optparse' option = { - my_name: `git config --global user.name`.chomp, - my_email: `git config --global user.email`.chomp, - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp + 'my_email' => `git config --global user.email`.chomp, + 'my_name' => `git config --global user.name`.chomp, + 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| - option[:my_email] = email + option['my_email'] = email end opts.on('-n name', 'my_name') do |name| - option[:my_name] = name + option['my_name'] = name end opts.on('-l lab', 'lab') do |lab| - option[:lab] = lab + option['lab'] = lab + end + + opts.on('-t', 'my_token') do + option['my_token'] = %x(uuidgen).chomp + end + + opts.on_tail('-h', 'show this message') do + puts opts + exit end end options.parse! -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +def build_my_info(option) + build_my_info = BuildMyInfo.new(option['my_email']) + info_es = build_my_info.search_my_info + info_es.update option + info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + + build_my_info.config_my_info(info_es) +end + +build_my_info(option) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/build_my_info_client: optimize
by Luan Shengde 03 Feb '21

03 Feb '21

cancel modify my_info enable user invoke each method alone. add search method: my_info Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 53 +++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 29 deletions(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..6d35e97 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -18,60 +18,55 @@ require_relative '../lib/es_client' # usage: # require_relative 'build_my_info_client' # -# build_my_info = BuildMyInfo.new(email, name, lab) -# build_my_info.config_my_info +# build_my_info = BuildMyInfo.new(my_info['email']) +# build_my_info.config_my_info(my_info) class BuildMyInfo - def initialize(my_email, my_name, lab, my_token = nil) - @lab = lab || 'nolab' - @my_token = my_token || %x(uuidgen).chomp - @my_info = { - 'my_email' => my_email, - 'my_name' => my_name, - 'my_token' => @my_token, - 'my_login_name' => nil, - 'my_commit_url' => nil, - 'my_ssh_pubkey' => [] - } + def initialize(my_email) + @my_email = my_email + @es = ESClient.new(index: 'accounts') end - def config_default_yaml + def search_my_info + @es.query_by_id(@my_email) || {} + end + + def store_account_info(my_info) + @es.put_source_by_id(@my_email, my_info) + end + + def config_default_yaml(my_info) default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir default_yaml_file = "#{default_yaml_dir}/account.yaml" FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file default_yaml_info = YAML.load_file(default_yaml_file) || {} - default_yaml_info['my_email'] = @my_info['my_email'] - default_yaml_info['my_name'] = @my_info['my_name'] - default_yaml_info['lab'] = @lab + default_yaml_info['my_email'] = my_info['my_email'] + default_yaml_info['my_name'] = my_info['my_name'] + default_yaml_info['lab'] = my_info['lab'] File.open(default_yaml_file, 'w') do |f| f.puts default_yaml_info.to_yaml end end - def config_lab_yaml + def config_lab_yaml(my_info) lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir - lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + lab_yaml_file = "#{lab_yaml_dir}/#{my_info['lab']}.yaml" FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file lab_yaml_info = YAML.load_file(lab_yaml_file) || {} - lab_yaml_info['my_token'] = @my_info['my_token'] + lab_yaml_info['my_token'] = my_info['my_token'] File.open(lab_yaml_file, 'w') do |f| f.puts lab_yaml_info.to_yaml end end - def store_account_info - es = ESClient.new(index: 'accounts') - es.put_source_by_id(@my_info['my_email'], @my_info) - end - - def config_my_info - config_default_yaml - config_lab_yaml - store_account_info + def config_my_info(my_info) + store_account_info(my_info) + config_default_yaml(my_info) + config_lab_yaml(my_info) end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/monitor-job: fix rm gem sources
by Wei Jihui 03 Feb '21

03 Feb '21

gem sources -c can not remove sources, use -r Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/monitor-job | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/monitor-job b/tests/monitor-job index 5fd2d73c..0f495ae2 100755 --- a/tests/monitor-job +++ b/tests/monitor-job @@ -18,7 +18,7 @@ set_scheduler_config() install_gem_pkgs() { - gem sources -c -a https://gems.ruby-china.com/ + gem sources -r http://rubygems.org -a https://gems.ruby-china.com/ gem install activesupport git rest-client elasticsearch } -- 2.23.0

1 0

[PATCH lkp-tests] tests/monitor-job: fix rm gem sources
by Wei Jihui 03 Feb '21

03 Feb '21

gem sources -c can not remove sources, use -r Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/monitor-job | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/monitor-job b/tests/monitor-job index 5fd2d73c..4a5d8562 100755 --- a/tests/monitor-job +++ b/tests/monitor-job @@ -18,7 +18,8 @@ set_scheduler_config() install_gem_pkgs() { - gem sources -c -a https://gems.ruby-china.com/ + gem sources -r http://rubygems.org + gem sources -a https://gems.ruby-china.com/ gem install activesupport git rest-client elasticsearch } -- 2.23.0

2 1

[PATCH compass-ci] delimiter: bisect run script catch the exceptions and exit -1
by Cao Xueliang 02 Feb '21

02 Feb '21

Git bisect tag the commit good/bad/skip according to the bisect run script exit code. Exit code 0 is good, 125 is skip, the others between 1 and 127 is bad. When the ruby script raise a exception exit code is 1, git bisect will tag the commit to bad, but we should abort the bisect process. So we catch the exceptions and exit -1 in bisect run script. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/find-commit/bisect_run_script.rb | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/delimiter/find-commit/bisect_run_script.rb b/src/delimiter/find-commit/bisect_run_script.rb index 4877d79..8cdf675 100755 --- a/src/delimiter/find-commit/bisect_run_script.rb +++ b/src/delimiter/find-commit/bisect_run_script.rb @@ -36,9 +36,13 @@ class GitBisectRun end end -job_id = ARGV[0] -error_id = ARGV[1] -work_dir = ARGV[2] - -run = GitBisectRun.new job_id, error_id, work_dir -run.git_bisect +begin + job_id = ARGV[0] + error_id = ARGV[1] + work_dir = ARGV[2] + + run = GitBisectRun.new job_id, error_id, work_dir + run.git_bisect +rescue + exit -1 +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add stats script for dockerfile os migrating
by Bai Jing 02 Feb '21

02 Feb '21

[input] ^[[91mError: Unable to find a match: supervisor ^[[91m/bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index e31934573..95e2a272b 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -2,7 +2,12 @@ while (line = STDIN.gets) line.chomp! - next unless line =~ /build.fail/ - - puts line + case line + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + end + when /\/bin\/sh: (.+): command not found/ + puts "sh.command-not-found.#{$1}: 1" + end end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 02 Feb '21

02 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..72762cf72f2f --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a disk which storing multiple rootfs, used for jobs which need boot from disk. + +Each of these rootfs corresponds to an logical volume, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume size: 10G + +4. more info can refer os_mount.md -- 2.23.0

1 0