January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/4-docker: set param value
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: action=reboot after: action=run-only [why] new param value more easy to understand. usage 1: ./buildall -> execute $container/build, install first-run and start script usage 2: ./buildall run-only -> just execute $continer/first-run, start script, no build, install script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- sparrow/7-systemd/cci-network | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..3c0b7b0 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then [ -x build ] && ./build [ -x install ] && ./install fi diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..0438935 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -14,4 +14,4 @@ $CCI_SRC/sparrow/2-network/cifs # --restart=always option is not absolutely reliable # container need to start its dependences firstly. -$CCI_SRC/sparrow/4-docker/buildall reboot +$CCI_SRC/sparrow/4-docker/buildall run-only -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..13cb7c0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + + # example: 2021-01-28 + day = t.strftime('%Y-%m-%d') + # example: 2021-01 + month = t.strftime('%Y-%m') + # example: 2021 + year = t.strftime('%Y') return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/defconfig: load config yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

Load "/etc/compass-ci/register/*.yaml", get register account info from cci defaults config. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index fd13312..f9b0824 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -9,6 +9,7 @@ def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', '/etc/compass-ci/accounts/*.yaml', + '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/auto-submit: bind mount volume
by Liu Yinsi 29 Jan '21

29 Jan '21

mount /etc/compass-ci/register, to get auto-submit account info from config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..79463f6 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] sbin/auto-submit: config account yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, lab, my_token from ES, then config yaml [why] store account info into ES, this part already done by container service register-accounts, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..d8c4b5a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,6 +7,8 @@ require 'bunny' require 'json' require 'yaml' require_relative '../lib/build_my_info_client' +require 'set' +require "#{ENV['CCI_SRC']}/container/defconfig" # receive message and auto submit job class AutoSubmit @@ -86,15 +88,30 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def my_email + names = Set.new %w[auto-submit] + + defaults = relevant_defaults(names) + defaults['auto-submit']['my_email'] +end + +def query_account_info + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 29 Jan '21

29 Jan '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

1 0

[PATCH compass-ci] doc: add support-os-os_arch-os_version.md
by Xu Xijian 29 Jan '21

29 Jan '21

This md document contains the currently supporting os in our server. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/support-os-os_arch-os_version.md diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md new file mode 100644 index 0000000..6532f92 --- /dev/null +++ b/doc/support-os-os_arch-os_version.md @@ -0,0 +1,22 @@ +``` +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, + 20.09 cifs/nfs, + mainline-dailybuild cifs/nfs, + 20.03-dailybuild cifs/nfs, + 20.09-dailybuild cifs/nfs, + 20.03-sp1-dailybuild cifs/nfs, + x86 20.03 cifs/nfs, + 20.09 cifs/nfs, +centos aarch64 7 container, + 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, + 8.1.1911 cifs/nfs, + 8 container, + x86 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, +debian aarch64 sid cifs/nfs, + x86 sid cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, +``` -- 2.23.0

3 4

[PATCH v3 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..2d2f3bf 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/:warning:|\.warning$/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/:error:|\.error$|\.exit_code\./i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..95503ed 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) - t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + t = Time.now.to_s + + # example: 2021-01-28 + day = t[0..9] + # example: 2021-01 + month = t[0..6] + # example: 2021 + year = t[0..3] return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker: pack lkp-$arch.cgz
by Liu Yinsi 29 Jan '21

29 Jan '21

Pack lkp-$arch.cgz by deploy script, it is convenient for users to submit jobs, it doesn't need to be executed by the user. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..014036b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -39,6 +39,7 @@ do_one() fi [ -x first-run ] && ./first-run [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run ) [ -n "$LKP_SRC" ] && log_info "finish build $container." started_containers+="$container_name" -- 2.23.0

2 2