- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/5] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 08 Feb '21

08 Feb '21

when execute `sparrow/4-docker/buildall run-only`, if caller script export env DOCKER_SKIP_REBUILD, then image already is exist, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 8 ++++++++ 2 files changed, 14 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..820147a 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + return unless ENV.include? 'DOCKER_SKIP_REBUILD' + + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..6f14890 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,11 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ -z $DOCKER_SKIP_REBUILD ] && return + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 +} -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/lkp: add hostname for lkp
by Wei Jihui 08 Feb '21

08 Feb '21

[error] test cluster task with initramfs, it will use hostname. /lkp/lkp/src/lib/job.sh: line 114: hostname: command not found Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 82c67184..945e29cf 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -6,3 +6,4 @@ gzip which rsync ntpdate +hostname -- 2.23.0

1 0

[PATCH compass-ci] container/send-mail: fix no package/constant
by Luan Shengde 08 Feb '21

08 Feb '21

fix no constant: from send-mail.rb:8:in `<main>' /c/compass-ci/lib/mail-post/email_init.rb:9:in `<top (required)>': uninitialized constant REDIS_HOST (NameError) fix no package: from send-mail.rb:12:in `<main>' /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- rufus-scheduler (LoadError) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-mail/Dockerfile | 2 +- container/send-mail/send-mail.rb | 4 ++++ container/send-mail/start | 1 + 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index b334b07..29f1bc6 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index 436736c..387df02 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,6 +5,10 @@ require 'mail' require 'sinatra' + +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + require "#{ENV['CCI_SRC']}/lib/mail-post" mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp diff --git a/container/send-mail/start b/container/send-mail/start index 95d797f..4af533d 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -11,6 +11,7 @@ docker_rm 'send-mail' names = Set.new %w[ LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ddb43fbd2 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,36 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" + has_cmd "apt-get" && installer="apt-get" + has_cmd "pacman" && installer="pacman" +} + mac_dep() { brew install ruby -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++++++++++----------- 1 file changed, 28 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ac2974ef1 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,44 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux + ;; + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux + ;; + zypper) + sudo "$installer" install -y ruby ruby-devel gcc gcc-c++ make git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + emerge) + sudo "$installer" ruby rubygems gcc make dev-vcs/git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" + has_cmd "apt-get" && installer="apt-get" + has_cmd "pacman" && installer="pacman" + has_cmd "zypper" && installer="zypper" + has_cmd "emerge" && installer="emerge" +} + mac_dep() { brew install ruby -- 2.23.0

1 1

[PATCH lkp-tests] lib/common.rb: build_pkg_error_id function add filtering rule
by Lin Jiaxin 07 Feb '21

07 Feb '21

Some software use tempfile when build. The directory is random, thus every error will be a new error_id. In fact, they are same. Now remove the random part except /tmp/lkp (makepkg argument: BUILDDIR). /tmp/gitstatus-build.BOpUcJeETC/libgit2/src/futils.c:1143:16: warning: comparison of integer expressions of different signedness: 'uint64_t' {aka 'const long unsigned int'} and 'long int' [-Wsign-compare] => libgit2/src/futils.c:warning:comparison-of-integer-expressions-of-different-signedness:'uint64_t'{aka'const-long-unsigned-int'}and'long-int'[-Wsign-compare] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/common.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/common.rb b/lib/common.rb index cc64eb17..8ba6694d 100755 --- a/lib/common.rb +++ b/lib/common.rb @@ -339,5 +339,6 @@ def build_pkg_error_id(line) line = line.tr(" ", "-") line = line.gsub(/[-_.,;:#!\[(]+$/, "") line = line.gsub(/([-_.,;:#!]){3,}/, ":") + line = line.gsub(/^\/tmp\/((?!lkp).)+?\//, "") line end -- 2.23.0

1 0

[PATCH v2 lkp-tests] makepkg: avoid some unnecessary stderr
by Lin Jiaxin 07 Feb '21

07 Feb '21

unnecessary stderr like this: ==> /tmp/stderr <== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M10 0 129 0 0 137 0 --:--:-- --:--:-- --:--:-- 137 ^M 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0^M100 255k 0 255k 0 0 110k 0 --:--:-- 0:00:02 --:--:-- 313k ==> WARNING: Skipping verification of source file PGP signatures. ==> /tmp/stderr <== fcgi-2.4.2.tar.gz ... Passed ==> /tmp/stderr <== Cloning into bare repository '//shadowsocks-libev-git'... ==> /tmp/stderr <== shadowsocks-libev-git ... Skipped shadowsocks-libev@.service ... Passed shadowsocks-libev-server@.service ... Passed shadowsocks-libev-redir@.service ... Passed shadowsocks-libev-tunnel@.service ... Passed Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- etc/makepkg.conf | 12 ++++++------ sbin/makepkg | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 5d73399d..b2beaedb 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,12 +8,12 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' - 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') +DLAGENTS=('file::/usr/bin/my_curl -sS -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -sS -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'rsync::/usr/bin/rsync -q --no-motd -z %u %o' + 'scp::/usr/bin/scp -q -C %u %o') # Other common tools: # /usr/bin/snarf diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..7a92ea71 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -566,7 +566,7 @@ download_git() { if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" - if ! git clone --mirror "$url" "$dir"; then + if ! git clone -q --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -580,7 +580,7 @@ download_git() { exit 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" - if ! git fetch --all -p; then + if ! git fetch -q --all -p; then # only warn on failure to allow offline builds warning "$(gettext "Failure while updating %s %s repo")" "${repo}" "git" fi @@ -609,13 +609,13 @@ extract_git() { if [[ -d "${dir##*/}" ]]; then updating=1 cd_safe "${dir##*/}" - if ! git fetch; then + if ! git fetch -q; then error "$(gettext "Failure while updating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 fi cd_safe "$srcdir" - elif ! git clone "$dir"; then + elif ! git clone -q "$dir"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -645,7 +645,7 @@ extract_git() { fi if [[ $ref != "origin/HEAD" ]] || (( updating )) ; then - if ! git checkout --force --no-track -B makepkg $ref; then + if ! git checkout -q --force --no-track -B makepkg $ref; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -1270,10 +1270,10 @@ verify_integrity_one() { local source_name=$1 integ=$2 expectedsum=$3 local file="$(get_filename "$source_name")" - printf ' %s ... ' "$file" >&2 + printf ' %s ... ' "$file" if [[ $expectedsum = 'SKIP' ]]; then - printf '%s\n' "$(gettext "Skipped")" >&2 + printf '%s\n' "$(gettext "Skipped")" return fi @@ -1285,7 +1285,7 @@ verify_integrity_one() { local realsum="$(openssl dgst -${integ} "$file")" realsum="${realsum##* }" if [[ ${expectedsum,,} = "$realsum" ]]; then - printf '%s\n' "$(gettext "Passed")" >&2 + printf '%s\n' "$(gettext "Passed")" else printf '%s\n' "$(gettext "FAILED")" >&2 return 1 @@ -1553,7 +1553,7 @@ check_source_integrity() { warning "$(gettext "Skipping verification of source file checksums.")" check_pgpsigs "$@" elif (( SKIPPGPCHECK )); then - warning "$(gettext "Skipping verification of source file PGP signatures.")" + # warning "$(gettext "Skipping verification of source file PGP signatures.")" check_checksums "$@" else check_checksums "$@" -- 2.23.0

4 5

[PATCH v3 compass-ci] doc/write-PKGBUILD: fix documentation errors
by Lin Jiaxin 07 Feb '21

07 Feb '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/write-PKGBUILD.en.md | 5 +++-- doc/manual/write-PKGBUILD.zh.md | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/manual/write-PKGBUILD.en.md b/doc/manual/write-PKGBUILD.en.md index c0c25e9..c259399 100644 --- a/doc/manual/write-PKGBUILD.en.md +++ b/doc/manual/write-PKGBUILD.en.md @@ -13,8 +13,9 @@ PKGBUILD includes variables and functions. - pkgrel: Mandatory. It indicates the release number of a software package. - arch: Mandatory. It indicates the architecture sequence of a software package. - depends: Optional. It indicates the name of the dependency package required for running a software test. -- makedepends: Optional. It indicates the list of files required for building a software package. -- source: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. +- makedepends: Optional. It indicates the name of the dependency package required for building a software package. +- source: Optional. It indicates the list of files required for building a software package. +- md5sums: Optional. It indicates the MD5 hash value of each source file to verify the integrity of the source file during the build process. ## Defining a Function diff --git a/doc/manual/write-PKGBUILD.zh.md b/doc/manual/write-PKGBUILD.zh.md index 414bfdc..ff53f0f 100644 --- a/doc/manual/write-PKGBUILD.zh.md +++ b/doc/manual/write-PKGBUILD.zh.md @@ -13,8 +13,9 @@ PKGBUILD包含两部分内容：变量和函数。 - pkgrel: 必须定义，表示软件包的发布号； - arch: 必须定义，表示软件包使用的架构序列； - depends: 可选字段，软件测试运行时需要的依赖包名称； -- makedepends: 可选字段，构建软件包时需要的文件列表； -- source: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 +- makedepends: 可选字段，构建软件包时需要的依赖包名称； +- source: 可选字段，构建软件包时需要的文件列表； +- md5sums: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 ## 定义函数 -- 2.23.0

2 2

[PATCH v3 compass-ci 5/5] sparrow/4-docker/buildall: prevent duplicate start
by Liu Yinsi 07 Feb '21

07 Feb '21

[why] when run container which has start depends, some containers will be start many times, so avoid repeated starting containers is necessary. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index b24eca5..0d1f161 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -53,10 +53,13 @@ do_one_run() local container=$1 local container_name=$(basename $container) - cd "$container" - [ -x first-run ] && ./first-run - [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] && ./run + mkdir $tmpdir/start_$container_name 2>/dev/null && + ( + cd "$container" + [ -x first-run ] && ./first-run + [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run + ) } tmpdir=$(mktemp -d) -- 2.23.0

1 0

[PATCH v3 compass-ci 4/5] sparrow/4-docker/buildall: optimise function log_info() calling
by Liu Yinsi 07 Feb '21

07 Feb '21

before: just execute $container/build script, echo "build $container" after: only execute $container/build -> `docker build -t`, echo "build $container" [why] when execute `sparrow/4-docker/buildall run-only`, will execute $container/build script, but if image already exist, actually will not build container, it will exit code "1", so only exit code "0", call log_info "finish build $container" Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 83ecc9f..b24eca5 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,13 +29,9 @@ do_one_build() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - - log_info "start build $container." - [ -x build ] && ./build + [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install - - log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0