- Compass-ci - mailweb.openeuler.org

git count
by Wu Fengguang 21 Jan '21

21 Jan '21

Xueliang is our code star, thanks Xueliang! 9 Cao Xueliang 8 Wu Zhende 7 Liu Yinsi 6 Wei Jihui 6 Ren Wen 4 Xu Xijian 4 Liu Shaofei 4 Zhang Yale 3 Yu Chuan 3 Lin Jiaxin 2 Bai Jing 2 Hu Xuejiao 2 Lu Weitao 2 Xiao Shenwei 2 Wang Chenglong 2 Li Yuanchao 1 Wu Fengguang 1 Wang Yong 1 Luan Shengde 1 Li Ping 1 Lu Kaiyi 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Cui Yili 0 Zhang Dewan 0 Shi Zhichao

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

On Tue, Jan 05, 2021 at 10:03:22AM +0800, Li Ping wrote: >[Usage]: >submit -m -l job.yaml > ==> lftp -c mirror http://SRV_HTTP_HOST:SRV_HTTP_PORT/RESULT_ROOT -l = lftp? "download results" can be more meaningful feature name. lftp is just one of the clients. Thanks, Fengguang

2 1

[PATCH lkp-tests] daptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

TO ALL >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' DON't use IP address. Use domain name instead. Thanks, Fengguang

2 1

[PATCH lkp-tests] adaptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/centos | 1 + distro/adaptation/debian | 1 + 2 files changed, 2 insertions(+) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index 2cfe3877..8fbae666 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -335,3 +335,4 @@ liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: openssh-client: openssh-clients +postgresql-server: postgresql diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] providers/libvirt: customizing logger format
by Xiao Shenwei 21 Jan '21

21 Jan '21

format: datetime-severity: message such as: 2021-01-20 16:04:03 +0800-INFO: xxxxxx Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..69061ca --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + @name = filename + if FileTest.exist?(filename) + File.delete(filename) + end + super(filename) + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _, msg| + "#{datetime}-#{severity}: #{msg}\n" + end + end +end -- 2.23.0

3 7

[PATCH compass-ci] lib/job.cr: use an independent interface for submit
by Wu Zhende 21 Jan '21

21 Jan '21

When we use Job.new, we need to do some extra processing only when we submit the job. Use interface "initialized?" to determine whether to submit a job. However, there is a vulnerability. If a job with all fields initialized is submitted, the job can be successfully submitted without checking the account. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 29 ++++++++++------------------- src/scheduler/submit_job.cr | 1 + 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..964c520 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -55,25 +55,6 @@ class Job @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new - - # init job with "-1", or use the original job_content["id"] - id = "-1" if "#{id}" == "" - - if initialized? - if @hash["id"] == "#{id}" - return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") - - check_account_info() - set_sshr_info() - return - end - end - - @hash["id"] = JSON::Any.new("#{id}") - - check_required_keys() - check_account_info() - set_defaults() end METHOD_KEYS = %w( @@ -155,6 +136,16 @@ class Job end end + def submit(id = nil) + # init job with "-1", or use the original job_content["id"] + id = "-1" if "#{id}" == "" + @hash["id"] = JSON::Any.new("#{id}") + + check_required_keys() + check_account_info() + set_defaults() + end + private def set_defaults append_init_field() set_docker_os() diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index bcb265f..1837cb1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -7,6 +7,7 @@ class Sched job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) + job.submit(job_content["id"]?) job["commit_date"] = get_commit_date(job) cluster_file = job["cluster"] -- 2.23.0

2 1

[PATCH v3 lkp-tests] sbin/submit: add -f option for uploading libvirt config
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt FILE's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.rb | 11 +++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc98..812f8c299 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -16,6 +16,17 @@ require 'set' require 'pp' require 'English' +# copied from compass-ci/container/defconfig.rb +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def restore(ah, copy) if ah.class == Hash ah.clear.merge!(copy) diff --git a/sbin/submit b/sbin/submit index 39cdcf576..1d56369ad 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) unless opt_upload_files.empty? + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

3 4

[PATCH v4 lkp-tests] sbin/submit: add option -i for submit
by Wei Jihui 21 Jan '21

21 Jan '21

submit -i is for add other yamls content to job.yaml [usage] case 1: submit -i 1.yaml -i 2.yaml job.yaml var priority: 1.yaml < 2.yaml < job.yaml case 2: submit -i 1.yaml job.yaml -i 2.yaml var priority: 1.yaml < job.yaml < 2.yaml case 3: submit -i 1.yaml job1.yaml -i 2.yaml job2.yaml -i 3.yaml var priority: 1.yaml < job1.yaml < 2.yaml < 3.yaml var priority: 1.yaml < 2.yaml < job2.yaml < 3.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 +++++++++++++---- sbin/submit | 30 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..640b25a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,8 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :default_yamls + attr_accessor :override_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +224,8 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @default_yamls = [] + @override_yamls = [] end def source_file_symkey(file) @@ -254,8 +258,11 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + + revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? + revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? + revise_hash(hash, @overrides, true) unless @overrides.empty? + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +293,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls(include_yamls) + include_hash = {} + include_hash[comment_to_symbol("#{include_yamls.join(' ')}")] = nil + include_hash.merge!(load_yaml_merge(include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..e8b17648 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -66,6 +71,7 @@ options = OptionParser.new do |opts| end +ARGV_CLONE = ARGV.clone options.parse!(ARGV) seen_job_yaml = false @@ -91,6 +97,19 @@ end job_ids = [] job_hash_list = [] +def prioritize_include_yaml(include_yamls, jobfile) + default_yamls, override_yamls = [], [] + jobfile_index = ARGV_CLONE.index(jobfile) + include_yamls.each do |yaml| + if ARGV_CLONE.index(yaml) < jobfile_index + default_yamls << yaml + else + override_yamls << yaml + end + end + return default_yamls, override_yamls +end + def find_jobfile(jobfile) return jobfile if File.exist?(jobfile) @@ -103,14 +122,25 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" ARGV.each do |jobfile| + default_yamls, override_yamls = prioritize_include_yaml(opt_include_yamls, jobfile) jobfile = find_jobfile(jobfile) jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.default_yamls = find_jobfiles(default_yamls) + jobs.override_yamls = find_jobfiles(override_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

1 0

[PATCH compass-ci] container/add-software: if cd failed, don't go on exec rm cmd
by Xu Xijian 21 Jan '21

21 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index aab659b..8a1d9ea 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -42,7 +42,7 @@ install_cmds=( addsoftware:auto bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages - cd /mnt/root + cd /mnt/root && rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} > /dev/null 2>&1 " ) -- 2.23.0

2 2