- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 100 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 90 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..f31771e5c 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,95 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' |grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + + # When running some plug-in, the file was not found + if a['msg'].match(/No file was found when using (.+)\./) + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + end + + # No such command + if a['msg'].match(/No such file or directory: b (.+) :/) + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + end + + # The URL of the RPM is invalid + if a['msg'].match(/Failure downloading (.+), /) + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + end + + # Failed to download the yum repodata + if a['msg'].match(/Failed to download metadata for repo (.+) :/) + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + end + + if a['msg'].match(/Failed to download packages: Cannot download (.+):/) + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + end + + # The task includes an option with an undefined variable. + if a['msg'].match(/: (.+) is undefined/) + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + end + + # The command runing error. + if a['msg'].match(/non-zero return code/) + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + end + + # Some service could not fand the requested host. + if a['msg'].match(/Could not find the requested service (.+): host/) + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + end + + # Unable to start some services + if a['msg'].match(/Unable to start service (.+):/) + puts "service.error.unable-to-star-service.#{$1}: 1" + puts "service.error.unable-to-star-service.#{$1}.message: #{ansible_stats_error}" + end + + # Some role in not sipported this system + if a['msg'].match(/(.+) is not supported/) + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + end + + # failed to create temporary content file: The read operation timed out. + if a['msg'].match(/The read operation timed out/) + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + # Failed to install some of the specified packages. + items.match(/No package (.+) available/) + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + + next if a['message'].nil? + + # Could not find or access ansible yaml. + if a['message'].to_s.match(/Could not find or access (.+) /) + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

3 4

[PATCH lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

2 1

[PATCH v3 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, the submodules cannot be downloaded from Z9/Crystal during building packages. [how] By appending --norevisedomain param, disable submodules to be downloaded from Z9/Crystal. Just download submodules from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..9c2ddc257 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -134,7 +134,7 @@ update_softlink() build_source_pkg() { local mark="true" - local pkg_args="-A --check --skippgpcheck" + local pkg_args="-A --check --skippgpcheck --norevisedomain" [ "${DISTRO}" == "archlinux" ] && { local mark="pacman" local pkg_args="${pkg_args} -s --needed --noconfirm" @@ -142,7 +142,11 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" \ + BUILDDIR="$TMP" \ + CARCH="$arch" \ + PKGEXT=".cgz" \ + CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] sbin/makepkg: fix whether to change domain of submodules
by Liu Shaofei 10 Feb '21

10 Feb '21

the param "--norevisedomain" indicates that all of submodules will not be changed domian name, and downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..6b5836251 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -89,6 +89,7 @@ SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 +NOREVISEDOMAIN=0 # Forces the pkgver of the current PKGBUILD. Used by the fakeroot call # when dealing with svn/cvs/etc PKGBUILDs. @@ -587,6 +588,16 @@ download_git() { fi } +# modify submodules address to local server address. +# (( NOREVISEDOMAIN )) is true, not replace submodules domain, and download it from remote end. +modify_gitmodules() { + (( NOREVISEDOMAIN )) && return + + if [[ -f ".gitmodules" ]] && [[ -n ${GIT_SERVER} ]]; then + git config --system url."git://${GIT_SERVER}/".insteadOf https:// + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3421,6 +3429,7 @@ usage() { printf -- "$(gettext " --skipinteg Do not perform any verification checks on source files")\n" printf -- "$(gettext " --skippgpcheck Do not verify source files with PGP signatures")\n" printf -- "$(gettext " --verifysource Download source files (if needed) and perform integrity checks")\n" + printf -- "$(gettext " --norevisedomain Do not change domain name when downloading submodules repository")\n" echo printf -- "$(gettext "These options can be passed to %s:")\n" "pacman" echo @@ -3462,7 +3471,7 @@ OPT_LONG=('allsource' 'check' 'clean' 'cleanbuild' 'config:' 'force' 'geninteg' 'help' 'holdver' 'ignorearch' 'install' 'key:' 'log' 'noarchive' 'nobuild' 'nocolor' 'nocheck' 'nodeps' 'noextract' 'noprepare' 'nosign' 'pkg:' 'repackage' 'rmdeps' 'sign' 'skipchecksums' 'skipinteg' 'skippgpcheck' 'source' 'syncdeps' - 'verifysource' 'version') + 'verifysource' 'version' 'norevisedomain') # Pacman Options OPT_LONG+=('asdeps' 'noconfirm' 'needed' 'noprogressbar') @@ -3514,6 +3523,7 @@ while true; do -s|--syncdeps) DEP_BIN=1 ;; -S|--source) SOURCEONLY=1 ;; --verifysource) VERIFYSOURCE=1 ;; + --norevisedomain) NOREVISEDOMAIN=1 ;; -h|--help) usage; exit 0 ;; # E_OK -V|--version) version; exit 0 ;; # E_OK -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: optimize skip rebuild image judge
by Wei Jihui 10 Feb '21

10 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/defconfig.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index aa7d2fa..602b498 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,8 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) - return if ENV['action'].nil? - return unless ENV['action'].include? 'run-only' + return if ENV['action'] != 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" end -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

error: hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. /dev/vdb did not recognize. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/makepkg: download submodules from local server by default
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, these submodules are downloaded from the local server, but the local server does not have submodules download address. So the submodules repository fails to be downloaded to test machine during the build. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..b592a2ccb 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -587,6 +587,17 @@ download_git() { fi } +# modify submodules address to local server address. +# [ "$DISABLE_DOMAIN_CHANGE" == "false" ] => replace submodules domain, and download it from local server +# [ "$DISABLE_DOMAIN_CHANGE" == "true" ] => not replace submodules domain, and download it from remote end. +modify_gitmodules() { + [ "$DISABLE_DOMAIN_CHANGE" == "false" ] || return + + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3668,6 +3676,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +DISABLE_DOMAIN_CHANGE=${DISABLE_DOMAIN_CHANGE:-false} if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then -- 2.23.0

2 3

[PATCH v2 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

DISABLE_DOMAIN_CHANGE parameter value is true, "sbin/makepkg" file will not replace domain name of submodules, the submodules will be downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> i# Your branch is ahead of 'origin/master' by 1 commit. --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..8c744d807 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -142,7 +142,7 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" DISABLE_DOMAIN_CHANGE=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

2 2

[PATCH compass-ci] src/scheduler/kernel_params.cr: no rootovl for local mount
by Xu Xijian 09 Feb '21

09 Feb '21

If the os_mount is "local", there is no need of overlay and it should be writable, so in the kernel params, delete "rootovl" for it and change "ro" to "rw" for it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index ed64bea..6fbe405 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -3,7 +3,9 @@ class Job private def kernel_common_params - return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" + common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" + return "#{common_params} rw" if "#{self.os_mount}" == "local" + return "#{common_params} rootovl ro" end private def kernel_custom_params -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: fix undefined method `includes' for nil:NilClass
by Xu Xijian 09 Feb '21

09 Feb '21

[Why] When "ENV['action']" is nil, it will output error in this line in defconfig.rb: return unless ENV['action'].include? 'run-only' [errlog] Traceback (most recent call last): 1: from ./build:15:in `<main>' /home/xxj/compass-ci/container/defconfig.rb:52:in `docker_skip_rebuild': undefined method `include?' for nil:NilClass (NoMethodError) Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 7dc9f33..aa7d2fa 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,6 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) + return if ENV['action'].nil? return unless ENV['action'].include? 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" -- 2.23.0

1 0