February 2021 - Compass-ci - mailweb.openeuler.org

Re: [PATCH lkp-tests 1/2] sbin/makepkg: fix whether to change domain of submodules
by Wu Fengguang 09 Mar '21

09 Mar '21

On Wed, Feb 10, 2021 at 02:36:57PM +0800, Liu Shaofei wrote: >the param "--norevisedomain" indicates that all of submodules will not >be changed domian name, and downloaded from remote end. spell: domian => domain >Signed-off-by: Liu Shaofei <370072077(a)qq.com> >--- > sbin/makepkg | 19 ++++++++++++++----- > 1 file changed, 14 insertions(+), 5 deletions(-) > >diff --git a/sbin/makepkg b/sbin/makepkg >index 7a92ea71b..41e19a513 100755 >--- a/sbin/makepkg >+++ b/sbin/makepkg >@@ -89,6 +89,7 @@ SIGNPKG='' > SPLITPKG=0 > SOURCEONLY=0 > VERIFYSOURCE=0 >+NOREVISEDOMAIN=0 You are adding an option to makepkg. To keep compatibility, if the option is not given, it should do nothing. So I suggest change it to REWRITEGITSERVER=0 And use option name rewritegitserver Thanks, Fengguang > # Forces the pkgver of the current PKGBUILD. Used by the fakeroot call > # when dealing with svn/cvs/etc PKGBUILDs. >@@ -587,6 +588,16 @@ download_git() { > fi > } > >+# modify submodules address to local server address. >+# (( NOREVISEDOMAIN )) is true, not replace submodules domain, and download it from remote end. >+modify_gitmodules() { >+ (( NOREVISEDOMAIN )) && return >+ >+ if [[ -f ".gitmodules" ]] && [[ -n ${GIT_SERVER} ]]; then >+ git config --system url."git://${GIT_SERVER}/".insteadOf https:// >+ fi >+} >+ > extract_git() { > local netfile=$1 > >@@ -623,10 +634,7 @@ extract_git() { > > cd_safe "${dir##*/}" > >- # modify submodules address to local server address. >- if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then >- echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig >- fi >+ modify_gitmodules > > local ref=origin/HEAD > if [[ -n $fragment ]]; then >@@ -3462,7 +3470,7 @@ OPT_LONG=('allsource' 'check' 'clean' 'cleanbuild' 'config:' 'force' 'geninteg' > 'help' 'holdver' 'ignorearch' 'install' 'key:' 'log' 'noarchive' 'nobuild' > 'nocolor' 'nocheck' 'nodeps' 'noextract' 'noprepare' 'nosign' 'pkg:' 'repackage' > 'rmdeps' 'sign' 'skipchecksums' 'skipinteg' 'skippgpcheck' 'source' 'syncdeps' >- 'verifysource' 'version') >+ 'verifysource' 'version' 'norevisedomain') > > # Pacman Options > OPT_LONG+=('asdeps' 'noconfirm' 'needed' 'noprogressbar') >@@ -3514,6 +3522,7 @@ while true; do > -s|--syncdeps) DEP_BIN=1 ;; > -S|--source) SOURCEONLY=1 ;; > --verifysource) VERIFYSOURCE=1 ;; >+ --norevisedomain) NOREVISEDOMAIN=1 ;; > > -h|--help) usage; exit 0 ;; # E_OK > -V|--version) version; exit 0 ;; # E_OK >-- >2.23.0 >

2 1

[PATCH compass-ci 1/2] sbin/upstream-config: add upstream-config file
by Li Yuanchao 02 Mar '21

02 Mar '21

To support more than one upstream repos warehouse Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/upstream-config | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 sbin/upstream-config diff --git a/sbin/upstream-config b/sbin/upstream-config new file mode 100644 index 0000000..e3b167a --- /dev/null +++ b/sbin/upstream-config @@ -0,0 +1,10 @@ +--- +upstreams: +- url: https://gitee.com/wu_fengguang/upstream-repos.git + location: upstream +- url: https://gitee.com/wu_fengguang/ansible-repos.git + location: ansible +- url: https://gitee.com/wu_fengguang/dockerfile-repos.git + location: dockerfile +- url: https://gitee.com/wu_fengguang/pkgbuild-repos.git + location: pkgbuild -- 2.23.0

2 3

[PATCH compass-ci 2/2] lib/git_mirror.rb: support more than one upstream repos warehouse
by Li Yuanchao 02 Mar '21

02 Mar '21

Support monitor repos in upstream-repos/ansible-repos/dockerfile-repos/pkgbuild-repos, they are just like upstream-repos. And which to monitor can be configured in file upstream-config. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 55 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 22c4e07..4652921 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -81,7 +81,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "/srv/git/#{fork_info['belong']}/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) @@ -107,6 +107,7 @@ class MirrorMain @defaults = {} @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") + clone_upstream_repo load_fork_info connection_init handle_webhook @@ -124,35 +125,37 @@ class MirrorMain @fork_stat[git_repo] = get_fork_stat(git_repo) end - def load_defaults(repodir) + def load_defaults(repodir, belong) defaults_file = "#{repodir}/DEFAULTS" return unless File.exist?(defaults_file) - defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") + defaults_key = repodir == "#{REPO_DIR}/#{belong}" ? belong : repodir.delete_prefix("#{REPO_DIR}/#{belong}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) - @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key], belong) end - def traverse_repodir(repodir) + def traverse_repodir(repodir, belong) if File.directory? repodir - load_defaults(repodir) + load_defaults(repodir, belong) entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' - traverse_repodir("#{repodir}/#{entry}") + traverse_repodir("#{repodir}/#{entry}", belong) end else return if File.dirname(repodir) == REPO_DIR - project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/#{belong}/") fork_name = File.basename(repodir) - load_repo_file(repodir, project, fork_name) + load_repo_file(repodir, project, fork_name, belong) end end def load_fork_info - traverse_repodir(REPO_DIR) + @upstreams['upstreams'].each do |repo| + traverse_repodir("#{REPO_DIR}/#{repo['location']}", repo['location']) + end end def create_workers @@ -221,14 +224,16 @@ end # main thread class MirrorMain - def load_repo_file(repodir, project, fork_name) + def load_repo_file(repodir, project, fork_name, belong) git_repo = "#{project}/#{fork_name}" git_info = YAML.safe_load(File.open(repodir)) return if git_info.nil? || git_info['url'].nil? @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) + @git_info[git_repo]['belong'] = belong + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) + fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -453,8 +458,8 @@ end # main thread class MirrorMain - def merge_defaults(object_key, object) - return object if object_key == 'default' + def merge_defaults(object_key, object, belong) + return object if object_key == belong defaults_key = File.dirname(object_key) while defaults_key != '.' @@ -462,8 +467,28 @@ class MirrorMain defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(object) if @defaults['default'] + return @defaults[belong].merge(object) if @defaults[belong] return object end + + def clone_upstream_repo + if File.exist?("#{ENV['CCI_SRC']}/sbin/upstream-config") + @upstreams = YAML.safe_load(File.open('../sbin/upstream-config')) + @upstreams['upstreams'].each do |repo| + url = get_url(repo['url']) + %x(git clone #{url} #{REPO_DIR}/#{repo['location']} 2>&1) + end + else + puts 'ERROR: No upstream-config file' + return -1 + end + end + + def get_url(url) + if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") + url = "/srv/git/#{url.delete_prefix('https://')}" + end + return url + end end -- 2.23.0

2 3

[PATCH v3 compass-ci 1/2] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 02 Mar '21

02 Mar '21

The install-iso job need scheduler response the ipxe_response value, when consume the install-iso job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..88db69e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH compass-ci] lib/git_mirror.rb: fix that unexpected file would cause a load error
by Li Yuanchao 01 Mar '21

01 Mar '21

Sometimes there would be some not yaml like file in upstream-repos, such as README.md or user's temporary file. Now we'll do a judge after load and before any other actions, if it's not repo file, just return. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 76cdd1b..22c4e07 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -223,7 +223,10 @@ end class MirrorMain def load_repo_file(repodir, project, fork_name) git_repo = "#{project}/#{fork_name}" - @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + git_info = YAML.safe_load(File.open(repodir)) + return if git_info.nil? || git_info['url'].nil? + + @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) -- 2.23.0

2 2

[PATCH compass-ci 1/2] container/defconfig.sh: accelerate es/logging-es startup time
by Liu Yinsi 01 Mar '21

01 Mar '21

before: es/logging-es startup time is 60s after: es/logging-es startup time is 10s [how] check es ready or not every 2s, if ok, just set index, rather than sleep 60s. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.sh | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/defconfig.sh b/container/defconfig.sh index c33a790..ee66ea5 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -28,9 +28,18 @@ docker_rm() docker rm -f $container } -set_es_indices() +check_es_ready() { - find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; + port=$1 + for num in {1..30} + do + curl -s localhost:$port > /dev/null + if [ $? == "0" ]; then + break + else + sleep 2 + fi + done } push_image() -- 2.23.0

4 7

[PATCH compass-ci 2/2] sparrow/4-docker/buildall: add $action value "start-only"
by Liu Yinsi 01 Mar '21

01 Mar '21

[usage] ./buildall start-only [how] before: ./buildall run-only: call build_depends(), check images exist or not, then call start_depends() after: ./buildall start-only: only call start_depends() [why] for some users locally deploy compass-ci, they will not to build images, and some users can't install lockfile-create command, only run containers, no need to check images exist or not, so only call start_depends(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c9edbe9..6619600 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -68,7 +68,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + [ "$action" != "start-only" ] && build_depends $dir start_depends $dir done -- 2.23.0

2 2

[PATCH compass-ci] container: add get_job_stats function in assist-result
by Cao Xueliang 01 Mar '21

01 Mar '21

We can expose assist-result host/port to the SUT, then the SUT can get the job stats. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 11 +++++++++++ container/assist-result/views/get_job_stats.rb | 10 ++++++++++ 2 files changed, 21 insertions(+) create mode 100644 container/assist-result/views/get_job_stats.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 077a427..02956aa 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/get_job_stats' require_relative './views/check_job_credible' configure do @@ -34,3 +35,13 @@ post '/check_job_credible' do return [200, {'credible' => result}.to_json] end + +get '/get_job_stats/:job_id' do + begin + result = get_job_stats(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_stats.rb b/container/assist-result/views/get_job_stats.rb new file mode 100644 index 0000000..d294df3 --- /dev/null +++ b/container/assist-result/views/get_job_stats.rb @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_stats(job_id) + content = ESQuery.new.query_by_id(job_id) + return content['stats'] +end -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..c9edbe9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,8 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + docker ps | grep -wq "ssh_r" + [ "$?" == "0" ] && [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 2

[PATCH compass-ci] container/scheduler: install util-linux to use uuidgen
by Wu Zhende 26 Feb '21

26 Feb '21

submit job.yaml in scheduler docker needs command "uuidgen" to generate submit_id Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 0ad813c..7fb8e35 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -8,7 +8,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'util-linux' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0