- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 01 Feb '21

01 Feb '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

2 1

[PATCH v1 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..93bc9ce --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline : #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: check host files in $lab repos
by Li Ping 01 Feb '21

01 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..12f1e05 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + get_lab_hosts_file() set_defaults() end @@ -382,6 +383,22 @@ class Job end end + private def get_lab_hosts_file + if !@hash["host_file"]? + lab_hosts_file_name = @hash["testbox"] + error_msg = "hosts file not exist: #{lab_hosts_file_name}" + lab_hosts_file = "#{ENV["CCI_REPOS"]}/lab-#{@hash["lab"]}/hosts/#{lab_hosts_file_name}" + if File.file?(lab_hosts_file) + host_hash = YAML.parse(File.read(lab_hosts_file)) + @hash.merge!(JSON.parse(host_hash.to_json).as_h) + else + raise error_msg + end + else + @hash.delete("host_file") + end + end + private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete get_lab_host_file
by Li Ping 01 Feb '21

01 Feb '21

[why] lab_host_files are on the server side and should be read on the server side. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 867c3d193..087ca226e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -354,6 +354,7 @@ class Job if hosts_file hwconfig = load_yaml(hosts_file, {}) @job[source_file_symkey(hosts_file)] = nil + @job["host_file"] = "#{hosts_file}" @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig @load_hosts_done = true end @@ -369,26 +370,14 @@ class Job return @job.include?('tbox_group') end - def get_lab_hosts_file - lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" - - if File.file?(lab_hosts_file) - lab_hosts_file - else - nil - end - end - def get_hosts_file - lab_hosts_file = get_lab_hosts_file - return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" - return hosts_file if File.file?(hosts_file) - - raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" + if File.file?(hosts_file) + hosts_file + else + nil + end end def include_files -- 2.23.0

1 0

[PATCH v1 compass-ci] providers/libvirt: check host resource and download file
by Xiao Shenwei 01 Feb '21

01 Feb '21

if the host machine does not have the capability of executing tasks, raise error message. if not, download dependent file based on response Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index b4f8b4c..a500c31 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -8,6 +8,7 @@ require 'digest/md5' require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" +require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" def create_logger(hostname) filename = "#{hostname}.log" @@ -51,10 +52,13 @@ def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? + resource.parse_response(response) + context.merge!(resource.info) sched_client.delete_mac2host(mac) sched_client.delete_host2queues(hostname) end -- 2.23.0

1 0

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2

[PATCH v5 compass-ci 2/2] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] providers/lib: introduce context.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] expand erb templates need a class to save context [how] Context class provides a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 01 Feb '21

01 Feb '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..2d2f3bf 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/:warning:|\.warning$/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/:error:|\.error$|\.exit_code\./i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH lkp-tests 3/3] distro/depends: add monitor-job depends
by Wei Jihui 30 Jan '21

30 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/monitor-job | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/monitor-job diff --git a/distro/depends/monitor-job b/distro/depends/monitor-job new file mode 100644 index 00000000..5bfafa87 --- /dev/null +++ b/distro/depends/monitor-job @@ -0,0 +1,3 @@ +git +ruby-dev +build-essential -- 2.23.0

1 0

[PATCH lkp-tests 2/3] jobs: add monitor-job.yaml
by Wei Jihui 30 Jan '21

30 Jan '21

[usage] add this to CCI_SRC/sbin/auto_submit.yaml, like: repo: - submit monitor-job.yaml os=... monitor_repo=x job_yaml=x job_testbox=x Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/monitor-job.yaml | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 jobs/monitor-job.yaml diff --git a/jobs/monitor-job.yaml b/jobs/monitor-job.yaml new file mode 100644 index 00000000..9b4d4d69 --- /dev/null +++ b/jobs/monitor-job.yaml @@ -0,0 +1,7 @@ +suite: monitor-job +category: functional + +monitor-job: + monitor_repo: + job_yaml: + job_testbox: -- 2.23.0

1 0

[PATCH lkp-tests 1/3] tests: add monitor-job
by Wei Jihui 30 Jan '21

30 Jan '21

monitor-job is for monitor user repos(monitor_repo), if user's repos has new commit, auto submit target job yaml(job_yaml). Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/monitor-job | 63 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100755 tests/monitor-job diff --git a/tests/monitor-job b/tests/monitor-job new file mode 100755 index 00000000..5fd2d73c --- /dev/null +++ b/tests/monitor-job @@ -0,0 +1,63 @@ +#!/bin/sh +# - monitor_repo +# - job_yaml +# - job_testbox + +. $LKP_SRC/lib/log.sh + +set_scheduler_config() +{ + mkdir -p /etc/compass-ci/defaults + cat > /etc/compass-ci/defaults/$lab.yaml <<-EOF + SCHED_HOST: $SCHED_HOST + SCHED_PORT: $SCHED_PORT + lab: $lab + my_email: $my_email + EOF +} + +install_gem_pkgs() +{ + gem sources -c -a https://gems.ruby-china.com/ + gem install activesupport git rest-client elasticsearch +} + +build_my_info() +{ + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git "$CCI_SRC" + ruby "$CCI_SRC"/sbin/build-my-info.rb -e "$my_email" -n "$my_name" -l "$lab" + + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/lkp-tests.git "$LKP_SRC" +} + +download_monitor_repo() +{ + if [ -n "$monitor_repo" ]; then + git clone git://"$LKP_SERVER"/"$monitor_repo" + else + log_error "monitor_repo is not specified" && exit 1 + fi + + test_code=${monitor_repo##*/} + cp -r "$test_code"/* $LKP_SRC/ +} + +submit_job() +{ + yaml=${job_yaml##*/} + "$LKP_SRC"/sbin/submit -a "$yaml" testbox="$job_testbox" +} + +main() +{ + local CCI_SRC=/c/compass-ci + local LKP_SRC=/c/lkp-tests + + install_gem_pkgs + set_scheduler_config + build_my_info + download_monitor_repo + submit_job +} + +main -- 2.23.0

1 0

[PATCH compass-ci] update progress
by Wang Chenglong 30 Jan '21

30 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/support-os-os_arch-os_version.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 6532f92..b354700 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,22 @@ ``` -os os_arch os_version supported-os_mount -openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, - 20.09 cifs/nfs, - mainline-dailybuild cifs/nfs, +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, ,initramfs + 20.09 cifs/nfs, ,initramfs + mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, 20.09 cifs/nfs, -centos aarch64 7 container, - 7.6.1810 cifs/nfs, - 7.8.2003 cifs/nfs, +centos aarch64 7 container + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, +debian aarch64 sid cifs/nfs, ,initfamfs x86 sid cifs/nfs, -archlinux aarch64 2020-11-12 cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot: make sure of consuming delimiter@localhost subqueue at first
by Cao Xueliang 29 Jan '21

29 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index f8d7caa..9e2898e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -75,13 +75,15 @@ class Sched end idle_queues = [] of String + delimiter_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" + delimiter_queues << "#{queue}/delimiter@localhost" end - all_queues = rand_queues(sub_queues) + idle_queues + all_queues = delimiter_queues + rand_queues(sub_queues) + idle_queues - return all_queues + return all_queues.uniq end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/apply-account: add how to test OSS project
by Wu Fengguang 29 Jan '21

29 Jan '21

Some people apply account by just adding his personal project's git url to upstream-repos. Which is not enough -- we only offer free account to projects that are in our test coverage. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 3e253d0..fda2eba 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -3,7 +3,7 @@ You'll need an account to submit jobs. The account is mainly for tracking your resource consumptions. We have limited pool of testboxes, so there have to be some usage control. -We offer free accounts to OSS (Open Source Software) contributors. +We offer free accounts to OSS (Open Source Software) contributors where the OSS project is in our test coverage. You can prove yourself as OSS contributor by providing "my oss commit" in the below email. We offer free accounts to our collaborators too. If you are one of our @@ -26,10 +26,15 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. + - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. + - my oss commit: a git commit URL that has your name and email. The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). + If your contributed OSS project is not in upstream-repos, you may refer to + [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + to add it to compass-ci testing. ## step 2: receive an email -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/install-tiny: skip steps to deploy compass-ci server
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: execute "./install-tiny", locally deploy compass-ci, can submit job, run containers services, run qemu test. after: execute "./install-tiny server reboot", locally deploy compass-ci server, just run containers servers. [why] some user who locally deploy compass-ci, the host which just run containers can only access intranet, so skip some steps that need to access internet when deploy, can submit job and run qemu test will in other machine, use $LKP_SRC/setup/simply-ci to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index cb47cf5..414c5bc 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -4,10 +4,19 @@ # For your reference only. # It's better to run the below scripts step by step. +export server=$1 +export action=$2 + +server_skip() +{ + local deploy_script=$1 + [ "$server" != "server" ] && ${deploy_script} +} + [[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) cd $CCI_SRC/sparrow || exit -0-package/install +server_skip 0-package/install 0-package/common 1-storage/tiny 1-storage/permission @@ -15,9 +24,9 @@ cd $CCI_SRC/sparrow || exit 2-network/iptables 2-network/nfs 2-network/cifs -3-code/git +server_skip 3-code/git 3-code/dev-env . /etc/profile.d/compass.sh -4-docker/buildall -5-build/ipxe +4-docker/buildall $action +server_skip 5-build/ipxe 7-systemd/systemd-setup -- 2.23.0

3 7

[PATCH v3 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sbin: add script for download os rootfs
by Liu Yinsi 29 Jan '21

29 Jan '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100644 index 0000000..d5ea328 --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +export os=$1 +export os_arch=$2 +export os_version=$3 + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 1

[PATCH v3 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/submit: README file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..9ce7c25 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,30 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit almost all the linux OS. +# Avoid much repetitive testing work to adapt. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Edit executable file: + + submit + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your localhost. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/submit: submit file
by Luan Shengde 29 Jan '21

29 Jan '21

executable file for container: subject it can be run as the real command submit, when you run it, it will calls the command 'submit' in the container add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..20fdf61 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/submit: build file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 29 Jan '21

29 Jan '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..fa5509d --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0

[PATCH compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 29 Jan '21

29 Jan '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add script for download os rootfs
by Liu Yinsi 29 Jan '21

29 Jan '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100644 index 0000000..3977178 --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=/${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir || exit + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

3 4

[PATCH compass-ci 3/4] container/submit: submit file
by Luan Shengde 29 Jan '21

29 Jan '21

executable file for the container it can be run as the real command submit, when you run it, it will calls the command 'submit' in the container add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..67a27d8 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

2 2

[PATCH compass-ci 2/4] container/submit: build file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 29 Jan '21

29 Jan '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0

[PATCH v3 compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 29 Jan '21

29 Jan '21

add output usage add option -h|--help set default option to -h|--help if no options added Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/{build-my-info.rb => build-my-info} | 12 ++++++++++++ 1 file changed, 12 insertions(+) rename sbin/{build-my-info.rb => build-my-info} (78%) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info similarity index 78% rename from sbin/build-my-info.rb rename to sbin/build-my-info index fd75f51..ca6a1de 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/build-my-info: optimize options
by Luan Shengde 29 Jan '21

29 Jan '21

add output usage add option -h|--help set default option to -h|--help if no options added Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..028fbac 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -14,6 +14,11 @@ option = { } options = OptionParser.new do |opts| + opts.banner = "Usage: ./build-my-info.rb -e email -n name -l lab\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| option[:my_email] = email end @@ -25,8 +30,15 @@ options = OptionParser.new do |opts| opts.on('-l lab', 'lab') do |lab| option[:lab] = lab end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end end +ARGV << '-h' if ARGV.empty? + options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -- 2.23.0

2 3

[PATCH v2 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 29 Jan '21

29 Jan '21

[why] when we need reboot a vm, we need get it's pid from hostname, then check job_id is the same as required. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_matrixes.rb: treatment of stddev < 1%
by Lu Weitao 29 Jan '21

29 Jan '21

[Why] if stddev < 1% | == 0 We display nothing for stddev column like bellow, it may confuse user. 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 -5.2% 262249.03 redis.get_avg_throughput_rps 61.47 +5.5% 64.83 redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% [How] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 < 1% -5.2% 262249.03 < 1% redis.get_avg_throughput_rps 61.47 < 1% +5.5% 64.83 < 1% redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7760c93..6c88384 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -72,7 +72,7 @@ def get_values(value_list, success) if length > 1 && average != 0 stddev_percent = ( standard_deviation(value_list, average, length) * 100 / average - ).to_i + ) end { average: average, stddev_percent: stddev_percent, min: sorted[0], max: sorted[-1], sorted: sorted } @@ -204,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -615,6 +615,8 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 + return " < 1% " if stddev_percent < 1 + percent_str = get_suitable_number_str( stddev_percent.abs, percent_width - 4, -- 2.23.0

1 0

[PATCH compass-ci 1/2] srv-http/start: add os images data download directory
by Cui Yili 29 Jan '21

29 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

3 4

[PATCH v4 compass-ci 3/3] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 2 +- container/result-cifs/start | 2 +- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 6 +++--- 8 files changed, 10 insertions(+), 15 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 0b9601a..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-cifs/start b/container/os-cifs/start index 09ae3d2..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-nfs/start b/container/os-nfs/start index 9539561..1f4ed42 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { log_error "kernel module nfs is missing (try: sudo modprobe nfs)" diff --git a/container/result-cifs/start b/container/result-cifs/start index de38ca2..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..8d6a85a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -5,7 +5,7 @@ CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" +source ${CCI_SRC}/lib/log.sh build_depends() { @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -41,7 +41,7 @@ do_one() [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container: delete redundant log error info
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: log_error "[ERROR] kernel module ..." after: log_error "kernel module ..." [why] log_error(): echo "$(date +'%F %T') [ERROR] -- $*" >&2 when call log_error(), already has complete log format, so delete redundant "[ERROR]" in parmas. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 4 ++-- container/result-cifs/start | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..0b9601a 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..09ae3d2 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..9539561 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -6,11 +6,11 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { - log_error "[ERROR] kernel module nfsd is missing (try: sudo modprobe nfsd)" + log_error "kernel module nfsd is missing (try: sudo modprobe nfsd)" exit 1 } diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..de38ca2 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib: add log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..626a6e4 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"[INFO] %F %T ${0##*/}: $*" +} + +log_debug() +{ + date +"[DEBUG] %F %T ${0##*/}: $*" +} + +log_warn() +{ + date +"[WARN] %F %T ${0##*/}: $*" >&2 +} + +log_error() +{ + date +"[ERROR] %F %T ${0##*/}: $*" >&2 +} -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 ++-- container/os-cifs/start | 4 ++-- container/os-nfs/start | 4 ++-- container/result-cifs/start | 4 ++-- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 8 files changed, 15 insertions(+), 20 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..7577647 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

2 4

[PATCH compass-ci] sparrow/4-docker: set param value
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: action=reboot after: action=run-only [why] new param value more easy to understand. usage 1: ./buildall -> execute $container/build, install first-run and start script usage 2: ./buildall run-only -> just execute $continer/first-run, start script, no build, install script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- sparrow/7-systemd/cci-network | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..3c0b7b0 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then [ -x build ] && ./build [ -x install ] && ./install fi diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..0438935 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -14,4 +14,4 @@ $CCI_SRC/sparrow/2-network/cifs # --restart=always option is not absolutely reliable # container need to start its dependences firstly. -$CCI_SRC/sparrow/4-docker/buildall reboot +$CCI_SRC/sparrow/4-docker/buildall run-only -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..13cb7c0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + + # example: 2021-01-28 + day = t.strftime('%Y-%m-%d') + # example: 2021-01 + month = t.strftime('%Y-%m') + # example: 2021 + year = t.strftime('%Y') return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/defconfig: load config yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

Load "/etc/compass-ci/register/*.yaml", get register account info from cci defaults config. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index fd13312..f9b0824 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -9,6 +9,7 @@ def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', '/etc/compass-ci/accounts/*.yaml', + '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/auto-submit: bind mount volume
by Liu Yinsi 29 Jan '21

29 Jan '21

mount /etc/compass-ci/register, to get auto-submit account info from config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..79463f6 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] sbin/auto-submit: config account yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, lab, my_token from ES, then config yaml [why] store account info into ES, this part already done by container service register-accounts, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..d8c4b5a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,6 +7,8 @@ require 'bunny' require 'json' require 'yaml' require_relative '../lib/build_my_info_client' +require 'set' +require "#{ENV['CCI_SRC']}/container/defconfig" # receive message and auto submit job class AutoSubmit @@ -86,15 +88,30 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def my_email + names = Set.new %w[auto-submit] + + defaults = relevant_defaults(names) + defaults['auto-submit']['my_email'] +end + +def query_account_info + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 29 Jan '21

29 Jan '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

1 0

[PATCH compass-ci] doc: add support-os-os_arch-os_version.md
by Xu Xijian 29 Jan '21

29 Jan '21

This md document contains the currently supporting os in our server. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/support-os-os_arch-os_version.md diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md new file mode 100644 index 0000000..6532f92 --- /dev/null +++ b/doc/support-os-os_arch-os_version.md @@ -0,0 +1,22 @@ +``` +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, + 20.09 cifs/nfs, + mainline-dailybuild cifs/nfs, + 20.03-dailybuild cifs/nfs, + 20.09-dailybuild cifs/nfs, + 20.03-sp1-dailybuild cifs/nfs, + x86 20.03 cifs/nfs, + 20.09 cifs/nfs, +centos aarch64 7 container, + 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, + 8.1.1911 cifs/nfs, + 8 container, + x86 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, +debian aarch64 sid cifs/nfs, + x86 sid cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, +``` -- 2.23.0

3 4

[PATCH v3 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..2d2f3bf 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/:warning:|\.warning$/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/:error:|\.error$|\.exit_code\./i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..95503ed 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) - t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + t = Time.now.to_s + + # example: 2021-01-28 + day = t[0..9] + # example: 2021-01 + month = t[0..6] + # example: 2021 + year = t[0..3] return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker: pack lkp-$arch.cgz
by Liu Yinsi 29 Jan '21

29 Jan '21

Pack lkp-$arch.cgz by deploy script, it is convenient for users to submit jobs, it doesn't need to be executed by the user. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..014036b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -39,6 +39,7 @@ do_one() fi [ -x first-run ] && ./first-run [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run ) [ -n "$LKP_SRC" ] && log_info "finish build $container." started_containers+="$container_name" -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow: move gem pkg into install script
by Liu Yinsi 29 Jan '21

29 Jan '21

Easy to call individually common script without internet Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 -- sparrow/0-package/install | 2 +- sparrow/install-tiny | 1 + 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..ccdc3f3 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch - grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer grep -q '^team:' /etc/group || groupadd team diff --git a/sparrow/0-package/install b/sparrow/0-package/install index 2b9a8f5..e01fc45 100755 --- a/sparrow/0-package/install +++ b/sparrow/0-package/install @@ -6,4 +6,4 @@ source /etc/os-release . $(dirname ${BASH_SOURCE[0]})/os/${ID} -. $(dirname ${BASH_SOURCE[0]})/common +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..cb47cf5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -8,6 +8,7 @@ cd $CCI_SRC/sparrow || exit 0-package/install +0-package/common 1-storage/tiny 1-storage/permission 2-network/br0 -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] lib: add log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..626a6e4 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"[INFO] %F %T ${0##*/}: $*" +} + +log_debug() +{ + date +"[DEBUG] %F %T ${0##*/}: $*" +} + +log_warn() +{ + date +"[WARN] %F %T ${0##*/}: $*" >&2 +} + +log_error() +{ + date +"[ERROR] %F %T ${0##*/}: $*" >&2 +} -- 2.23.0

3 4

[PATCH v2 compass-ci] sparrow/0-package: set umask 002
by Liu Yinsi 29 Jan '21

29 Jan '21

Insure programs have necessary permission to work normally. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..6b4a834 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -27,6 +27,12 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF +cat >> /etc/bashrc <<EOF +umask 002 +EOF + +source /etc/bashrc + server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..9e0c671 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/warning/i) && !stat.match(/nr_warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/error|\.exit_code\./i) && !stat.match(/nr_error/i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

2 1

[PATCH v3 lab-z9] modify the lab-z9/allot/idle
by Zhang Dewan 28 Jan '21

28 Jan '21

dewan@crystal ~/128/lab-z9/allot% tree . . └── idle ├── taishan200-2280-2s48p-256g--a11 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a12 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a13 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a14 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a15 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a16 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a17 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a18 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a19 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a20 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a60 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--a61 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--a62 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--multi-docker │ └── multi-docker.yaml └── taishan200-2280-2s48p-256g--multi-qemu └── multi-qemu.yaml Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a11 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a12 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a13 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a14 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a15 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a16 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a17 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a18 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a19 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a20 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a21 | 1 - allot/idle/taishan200-2280-2s48p-256g--a22 | 1 - allot/idle/taishan200-2280-2s48p-256g--a24 | 1 - allot/idle/taishan200-2280-2s48p-256g--a25 | 1 - allot/idle/taishan200-2280-2s48p-256g--a26 | 1 - allot/idle/taishan200-2280-2s48p-256g--a27 | 1 - allot/idle/taishan200-2280-2s48p-256g--a28 | 1 - allot/idle/taishan200-2280-2s48p-256g--a29 | 1 - allot/idle/taishan200-2280-2s48p-256g--a30 | 1 - allot/idle/taishan200-2280-2s48p-256g--a31 | 1 - allot/idle/taishan200-2280-2s48p-256g--a32 | 1 - allot/idle/taishan200-2280-2s48p-256g--a33 | 1 - allot/idle/taishan200-2280-2s48p-256g--a34 | 1 - allot/idle/taishan200-2280-2s48p-256g--a35 | 1 - allot/idle/taishan200-2280-2s48p-256g--a60 | 1 + allot/idle/taishan200-2280-2s48p-256g--a61 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a62 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a63 | 1 - allot/idle/taishan200-2280-2s48p-256g--a64 | 1 - allot/idle/taishan200-2280-2s48p-256g--a65 | 1 - allot/idle/taishan200-2280-2s48p-256g--a66 | 1 - allot/idle/taishan200-2280-2s48p-256g--a67 | 1 - allot/idle/taishan200-2280-2s48p-256g--a68 | 1 - allot/idle/taishan200-2280-2s48p-256g--a69 | 1 - allot/idle/taishan200-2280-2s48p-256g--a70 | 1 - allot/idle/taishan200-2280-2s48p-256g--a71 | 1 - .../multi-docker.yaml | 17 +++++++++++++++++ .../multi-qemu.yaml | 0 38 files changed, 30 insertions(+), 35 deletions(-) delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a21 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a22 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a24 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a25 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a26 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a27 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a28 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a29 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a30 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a31 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a32 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a33 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a34 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a35 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a60 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a71 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml rename allot/idle/{taishan200-2280-2s48p-256g--a60 => taishan200-2280-2s48p-256g--multi-qemu}/multi-qemu.yaml (100%) diff --git a/allot/idle/taishan200-2280-2s48p-256g--a11 b/allot/idle/taishan200-2280-2s48p-256g--a11 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a11 +++ b/allot/idle/taishan200-2280-2s48p-256g--a11 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a12 b/allot/idle/taishan200-2280-2s48p-256g--a12 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a12 +++ b/allot/idle/taishan200-2280-2s48p-256g--a12 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a13 b/allot/idle/taishan200-2280-2s48p-256g--a13 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a13 +++ b/allot/idle/taishan200-2280-2s48p-256g--a13 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a14 b/allot/idle/taishan200-2280-2s48p-256g--a14 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a14 +++ b/allot/idle/taishan200-2280-2s48p-256g--a14 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a15 b/allot/idle/taishan200-2280-2s48p-256g--a15 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a15 +++ b/allot/idle/taishan200-2280-2s48p-256g--a15 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a16 b/allot/idle/taishan200-2280-2s48p-256g--a16 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a16 +++ b/allot/idle/taishan200-2280-2s48p-256g--a16 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a17 b/allot/idle/taishan200-2280-2s48p-256g--a17 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a17 +++ b/allot/idle/taishan200-2280-2s48p-256g--a17 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a18 b/allot/idle/taishan200-2280-2s48p-256g--a18 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a18 +++ b/allot/idle/taishan200-2280-2s48p-256g--a18 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a19 b/allot/idle/taishan200-2280-2s48p-256g--a19 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a19 +++ b/allot/idle/taishan200-2280-2s48p-256g--a19 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a20 b/allot/idle/taishan200-2280-2s48p-256g--a20 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a20 +++ b/allot/idle/taishan200-2280-2s48p-256g--a20 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a21 b/allot/idle/taishan200-2280-2s48p-256g--a21 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a21 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a22 b/allot/idle/taishan200-2280-2s48p-256g--a22 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a22 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a24 b/allot/idle/taishan200-2280-2s48p-256g--a24 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a24 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a25 b/allot/idle/taishan200-2280-2s48p-256g--a25 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a25 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a26 b/allot/idle/taishan200-2280-2s48p-256g--a26 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a26 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a27 b/allot/idle/taishan200-2280-2s48p-256g--a27 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a27 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a28 b/allot/idle/taishan200-2280-2s48p-256g--a28 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a28 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a29 b/allot/idle/taishan200-2280-2s48p-256g--a29 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a29 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a30 b/allot/idle/taishan200-2280-2s48p-256g--a30 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a30 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a31 b/allot/idle/taishan200-2280-2s48p-256g--a31 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a31 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a32 b/allot/idle/taishan200-2280-2s48p-256g--a32 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a32 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a33 b/allot/idle/taishan200-2280-2s48p-256g--a33 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a33 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a34 b/allot/idle/taishan200-2280-2s48p-256g--a34 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a34 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a35 b/allot/idle/taishan200-2280-2s48p-256g--a35 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a35 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60 b/allot/idle/taishan200-2280-2s48p-256g--a60 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a60 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a61 b/allot/idle/taishan200-2280-2s48p-256g--a61 index 479a75a..3c3d0c3 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a61 +++ b/allot/idle/taishan200-2280-2s48p-256g--a61 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a62 b/allot/idle/taishan200-2280-2s48p-256g--a62 index 479a75a..3c3d0c3 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a62 +++ b/allot/idle/taishan200-2280-2s48p-256g--a62 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a63 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a64 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a65 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a66 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a67 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a68 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a69 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a70 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a71 b/allot/idle/taishan200-2280-2s48p-256g--a71 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a71 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml new file mode 100644 index 0000000..8e42208 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml @@ -0,0 +1,17 @@ +suite: multi-docker +category: functional +simplify-ci: +runtime: 100d +multi-docker-0: + nr_container: 20 + tbox_group: dc-1g +multi-docker-1: + nr_container: 10 + tbox_group: dc-2g +multi-docker-2: + nr_container: 5 + tbox_group: dc-4g +multi-docker-3: + nr_container: 20 + tbox_group: dc-8g +sleep: diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml rename to allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] scheduler/redis_client.cr: add singleton pattern
by Wu Zhende 28 Jan '21

28 Jan '21

Singleton pattern will only generate an instance. The Redis pool needs to be initialized only once and the connection is obtained from the pool. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/redis_client.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index 1253108..a8d9dcb 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -6,12 +6,17 @@ require "redis" require "./constants" require "../lib/job" +require "singleton" class Redis::Client class_property :client HOST = (ENV.has_key?("REDIS_HOST") ? ENV["REDIS_HOST"] : JOB_REDIS_HOST) PORT = (ENV.has_key?("REDIS_PORT") ? ENV["REDIS_PORT"] : JOB_REDIS_PORT).to_i32 + def self.instance + Singleton::Of(self).instance + end + def initialize(host = HOST, port = PORT) @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) end -- 2.23.0

2 1

[PATCH compass-ci 3/3] lib/sched.cr: redis use singleton pattern
by Wu Zhende 28 Jan '21

28 Jan '21

[Why] A redis pool is created for each HTTP request. This is wrong. This can cause too many connections to be established. All HTTP requests should share a redis pool. [How] Use singleton pattern Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f46d97c..b90759f 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -32,7 +32,7 @@ class Sched def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new - @redis = Redis::Client.new + @redis = Redis::Client.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new @env = env -- 2.23.0

1 0

[PATCH compass-ci 1/3] crystal-shards/shard.yml: add singleton.cr
by Wu Zhende 28 Jan '21

28 Jan '21

In order to use the singleton pattern Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/crystal-shards/shard.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/crystal-shards/shard.yml b/container/crystal-shards/shard.yml index 65a7622..44d175d 100644 --- a/container/crystal-shards/shard.yml +++ b/container/crystal-shards/shard.yml @@ -30,3 +30,5 @@ dependencies: github: anykeyh/json_on_steroids rate_limiter: github: z64/rate_limiter + singleton: + github: waterlink/singleton.cr -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] expand erb templates need a class to save context [how] Context class provides a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 28 Jan '21

28 Jan '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 2 +- distro/archlinux | 2 +- distro/common | 12 ++++++++++++ distro/debian | 2 +- tests/cci-depends | 9 ++------- 5 files changed, 17 insertions(+), 10 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 598243d8e..5c234ada6 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -80,7 +80,7 @@ yum_download() yum_pack() { local target_dir=/opt/rpms - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_rpms=$(find /var/ -type f -name "*.rpm") diff --git a/distro/archlinux b/distro/archlinux index 3942080a6..3d576555c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -50,7 +50,7 @@ pacman_download() pacman_pack() { local target_dir=/opt/pkgs - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_pkgs=$(find_pacman_package) diff --git a/distro/common b/distro/common index 1fd7df4e8..d1a3a5447 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,15 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + + cd "$pack_to" + [ -f ${BM_NAME}.cgz ] || return + rm ${BM_NAME}.cgz + mv ${BM_NAME}_$date.cgz ${benchmark}_$date.cgz + + ln -sf ${benchmark}_$date.cgz ${benchmark}.cgz +} diff --git a/distro/debian b/distro/debian index 8821e3ddd..b07d395b6 100755 --- a/distro/debian +++ b/distro/debian @@ -99,7 +99,7 @@ fixup_perf() pack() { - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") [[ $(type -t fixup_$BM_NAME) = 'function' ]] && fixup_$BM_NAME diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..83c330c --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"%F %T $* [INFO]" +} + +log_debug() +{ + date +"%F %T $* [DEBUG]" +} + +log_warn() +{ + date +"%F %T $* [WARN]" >&2 +} + +log_error() +{ + date +"%F %T $* [ERROR]" >&2 +} -- 2.23.0

2 5

[PATCH lkp-tests 3/3] sbin/install-dependencies: replace gem sources
by Liu Yinsi 28 Jan '21

28 Jan '21

Encapsulating functions install_gem_pkg for easy gem sources replacement Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index f2a9155f6..8bd1a1ddb 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,6 +6,13 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh # choose install function base on DISTRIBUTION + +install_gem_pkg() +{ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + sudo gem install -f git activesupport rest-client faye-websocket +} + linux_dep() { source /etc/os-release @@ -13,7 +20,7 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket git activesupport rest-client + install_gem_pkg ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -21,7 +28,7 @@ linux_dep() else sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp fi - sudo gem install -f git activesupport rest-client faye-websocket + install_gem_pkg ;; *) echo "$ID not support! please install dependencies manually." @@ -34,7 +41,7 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - sudo gem install -f git activesupport rest-client + install_gem_pkg } run() -- 2.23.0

1 0

[PATCH lkp-tests 2/3] sbin/install-dependencies: classify gem pkg
by Liu Yinsi 28 Jan '21

28 Jan '21

Install ruby pkg by gem install, not by apt-get install. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index aa069bfcd..f2a9155f6 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -12,8 +12,8 @@ linux_dep() case $ID in ubuntu|debian) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket + sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ + sudo gem install -f faye-websocket git activesupport rest-client ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then -- 2.23.0

1 0

[PATCH lkp-tests 1/3] sbin/install-dependencies: fix inconsistent gem command
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..aa069bfcd 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -13,7 +13,7 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client ruby-dev libssl-dev gcc g++ - sudo gem install faye-websocket + sudo gem install -f faye-websocket ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -34,7 +34,7 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - gem install git activesupport rest-client + sudo gem install -f git activesupport rest-client } run() -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: delete unnecessary update of es
by Li Yuanchao 28 Jan '21

28 Jan '21

It does not need to update es when init, cause nothing changes yet. And update es here would make git-mirror start slowly. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index a111e51..2090f6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -214,7 +214,6 @@ class MirrorMain @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] fork_stat_init(git_repo) - es_repo_update(git_repo) @priority_queue.push git_repo, @priority @priority += 1 end -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: change ip url to domain name url
by Li Ping 28 Jan '21

28 Jan '21

when the external ip address of host is the same as the srv_http_host, the host can not use the url to get result. --- lib/monitor.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 9091c92a9..82e48c3e6 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,9 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" + url = "http://api.compass-ci.openeuler.org:11300#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" end -- 2.23.0

2 2

[PATCH v2 lab-z9] modify lab-z9/allot/idle, and add queue to the testbox about multi-qemu and multi-docker
by Zhang Dewan 28 Jan '21

28 Jan '21

dewan@crystal ~/tmp/lab-z9/allot/idle% tree . . ├── taishan200-2280-2s48p-256g--multi-docker │ ├── taishan200-2280-2s48p-256g--a11 │ ├── taishan200-2280-2s48p-256g--a12 │ ├── taishan200-2280-2s48p-256g--a13 │ ├── taishan200-2280-2s48p-256g--a14 │ └── taishan200-2280-2s48p-256g--a15 └── taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--a60 ├── taishan200-2280-2s48p-256g--a61 └── taishan200-2280-2s48p-256g--a62 2 directories, 8 files Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a11 | 1 - allot/idle/taishan200-2280-2s48p-256g--a12 | 1 - allot/idle/taishan200-2280-2s48p-256g--a13 | 1 - allot/idle/taishan200-2280-2s48p-256g--a14 | 1 - allot/idle/taishan200-2280-2s48p-256g--a15 | 1 - allot/idle/taishan200-2280-2s48p-256g--a16 | 1 - allot/idle/taishan200-2280-2s48p-256g--a17 | 1 - allot/idle/taishan200-2280-2s48p-256g--a18 | 1 - allot/idle/taishan200-2280-2s48p-256g--a19 | 1 - allot/idle/taishan200-2280-2s48p-256g--a20 | 1 - allot/idle/taishan200-2280-2s48p-256g--a21 | 1 - allot/idle/taishan200-2280-2s48p-256g--a22 | 1 - allot/idle/taishan200-2280-2s48p-256g--a24 | 1 - allot/idle/taishan200-2280-2s48p-256g--a25 | 1 - allot/idle/taishan200-2280-2s48p-256g--a26 | 1 - allot/idle/taishan200-2280-2s48p-256g--a27 | 1 - allot/idle/taishan200-2280-2s48p-256g--a28 | 1 - allot/idle/taishan200-2280-2s48p-256g--a29 | 1 - allot/idle/taishan200-2280-2s48p-256g--a30 | 1 - allot/idle/taishan200-2280-2s48p-256g--a31 | 1 - allot/idle/taishan200-2280-2s48p-256g--a32 | 1 - allot/idle/taishan200-2280-2s48p-256g--a33 | 1 - allot/idle/taishan200-2280-2s48p-256g--a34 | 1 - allot/idle/taishan200-2280-2s48p-256g--a35 | 1 - .../multi-qemu.yaml | 18 -------------- allot/idle/taishan200-2280-2s48p-256g--a61 | 1 - allot/idle/taishan200-2280-2s48p-256g--a62 | 1 - allot/idle/taishan200-2280-2s48p-256g--a63 | 1 - allot/idle/taishan200-2280-2s48p-256g--a64 | 1 - allot/idle/taishan200-2280-2s48p-256g--a65 | 1 - allot/idle/taishan200-2280-2s48p-256g--a66 | 1 - allot/idle/taishan200-2280-2s48p-256g--a67 | 1 - allot/idle/taishan200-2280-2s48p-256g--a68 | 1 - allot/idle/taishan200-2280-2s48p-256g--a69 | 1 - allot/idle/taishan200-2280-2s48p-256g--a70 | 1 - allot/idle/taishan200-2280-2s48p-256g--a71 | 1 - .../taishan200-2280-2s48p-256g--a11 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a12 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a13 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a14 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a15 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a60 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a61 | 24 +++++++++++++++++++ .../taishan200-2280-2s48p-256g--a62 | 24 +++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a11 | 2 ++ hosts/taishan200-2280-2s48p-256g--a12 | 2 ++ hosts/taishan200-2280-2s48p-256g--a13 | 2 ++ hosts/taishan200-2280-2s48p-256g--a14 | 2 ++ hosts/taishan200-2280-2s48p-256g--a15 | 2 ++ hosts/taishan200-2280-2s48p-256g--a60 | 2 ++ hosts/taishan200-2280-2s48p-256g--a61 | 2 ++ hosts/taishan200-2280-2s48p-256g--a62 | 2 ++ 52 files changed, 208 insertions(+), 53 deletions(-) delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a11 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a12 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a13 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a14 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a15 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a16 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a17 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a18 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a19 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a20 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a21 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a22 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a24 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a25 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a26 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a27 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a28 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a29 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a30 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a31 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a32 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a33 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a34 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a35 delete mode 100644 allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a61 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a62 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a71 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a11 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a12 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a13 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a14 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a15 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a60 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a61 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a62 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a11 b/allot/idle/taishan200-2280-2s48p-256g--a11 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a11 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a12 b/allot/idle/taishan200-2280-2s48p-256g--a12 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a12 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a13 b/allot/idle/taishan200-2280-2s48p-256g--a13 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a13 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a14 b/allot/idle/taishan200-2280-2s48p-256g--a14 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a14 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a15 b/allot/idle/taishan200-2280-2s48p-256g--a15 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a15 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a16 b/allot/idle/taishan200-2280-2s48p-256g--a16 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a16 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a17 b/allot/idle/taishan200-2280-2s48p-256g--a17 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a17 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a18 b/allot/idle/taishan200-2280-2s48p-256g--a18 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a18 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a19 b/allot/idle/taishan200-2280-2s48p-256g--a19 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a19 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a20 b/allot/idle/taishan200-2280-2s48p-256g--a20 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a20 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a21 b/allot/idle/taishan200-2280-2s48p-256g--a21 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a21 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a22 b/allot/idle/taishan200-2280-2s48p-256g--a22 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a22 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a24 b/allot/idle/taishan200-2280-2s48p-256g--a24 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a24 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a25 b/allot/idle/taishan200-2280-2s48p-256g--a25 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a25 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a26 b/allot/idle/taishan200-2280-2s48p-256g--a26 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a26 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a27 b/allot/idle/taishan200-2280-2s48p-256g--a27 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a27 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a28 b/allot/idle/taishan200-2280-2s48p-256g--a28 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a28 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a29 b/allot/idle/taishan200-2280-2s48p-256g--a29 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a29 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a30 b/allot/idle/taishan200-2280-2s48p-256g--a30 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a30 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a31 b/allot/idle/taishan200-2280-2s48p-256g--a31 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a31 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a32 b/allot/idle/taishan200-2280-2s48p-256g--a32 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a32 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a33 b/allot/idle/taishan200-2280-2s48p-256g--a33 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a33 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a34 b/allot/idle/taishan200-2280-2s48p-256g--a34 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a34 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a35 b/allot/idle/taishan200-2280-2s48p-256g--a35 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a35 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml deleted file mode 100644 index 513b631..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml +++ /dev/null @@ -1,18 +0,0 @@ -suite: multi-qemu -category: functional - -os: openeuler -os_version: 20.03 -os_arch: aarch64 -os_mount: initramfs -simplify-ci: -multi-qemu-0: - nr_vm: 20 - tbox_group: vm-2p8g -multi-qemu-1: - nr_vm: 2 - tbox_group: vm-2p4g -multi-qemu-2: - nr_vm: 2 - tbox_group: vm-1p1g -sleep: 100d diff --git a/allot/idle/taishan200-2280-2s48p-256g--a61 b/allot/idle/taishan200-2280-2s48p-256g--a61 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a61 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a62 b/allot/idle/taishan200-2280-2s48p-256g--a62 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a62 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a63 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a64 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a65 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a66 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a67 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a68 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a69 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a70 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a71 b/allot/idle/taishan200-2280-2s48p-256g--a71 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a71 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a11 b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a11 new file mode 100644 index 0000000..0a7f15e --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a11 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b8891431-part1 + - /dev/disk/by-id/scsi-350000399b8890e09 + - /dev/disk/by-id/scsi-350000399b8890cd1 +rootfs_disk: + - /dev/disk/by-id/scsi-350000399b8727929 +mac_addr: + - cc:05:77:ab:97:62 + - cc:05:77:ab:97:63 + - cc:05:77:ab:97:64 + - cc:05:77:ab:97:65 + - cc:05:77:ab:97:66 + - cc:05:77:ab:97:67 + - cc:05:77:ab:97:68 + - cc:05:77:ab:97:69 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.1 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a12 b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a12 new file mode 100644 index 0000000..0ac46fe --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a12 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500c276c347 + - /dev/disk/by-id/scsi-35000c500c276eb87 + - /dev/disk/by-id/scsi-35000c500c276b6ff +rootfs_disk: + - /dev/disk/by-id/scsi-35000c500c276f47b +mac_addr: + - 44:67:47:97:22:37 + - 44:67:47:97:22:38 + - 44:67:47:97:22:39 + - 44:67:47:97:22:3a + - 44:67:47:97:22:3b + - 44:67:47:97:22:3c + - 44:67:47:97:22:3d + - 44:67:47:97:22:3e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.2 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a13 b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a13 new file mode 100644 index 0000000..5cc2506 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a13 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500c2799bb3 + - /dev/disk/by-id/scsi-35000c500c278b18b + - /dev/disk/by-id/scsi-35000c500c279f96b +rootfs_disk: + - /dev/disk/by-id/scsi-35000c500c2789b4b +mac_addr: + - 44:67:47:97:2b:bf + - 44:67:47:97:2b:c0 + - 44:67:47:97:2b:c1 + - 44:67:47:97:2b:c2 + - 44:67:47:97:2b:c3 + - 44:67:47:97:2b:c4 + - 44:67:47:97:2b:c5 + - 44:67:47:97:2b:c6 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.3 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a14 b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a14 new file mode 100644 index 0000000..ce617ad --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a14 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88b4475 + - /dev/disk/by-id/scsi-350000399b88b4371 + - /dev/disk/by-id/scsi-350000399b88b432d +rootfs_disk: + - /dev/disk/by-id/scsi-350000399b88b4531 +mac_addr: + - 44:67:47:c9:e9:c2 + - 44:67:47:c9:e9:c3 + - 44:67:47:c9:e9:c4 + - 44:67:47:c9:e9:c5 + - 44:67:47:c9:e9:c6 + - 44:67:47:c9:e9:c7 + - 44:67:47:c9:e9:c8 + - 44:67:47:c9:e9:c9 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.4 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a15 b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a15 new file mode 100644 index 0000000..ce2079e --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/taishan200-2280-2s48p-256g--a15 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b841fff5 + - /dev/disk/by-id/scsi-350000399b841fcb9 + - /dev/disk/by-id/scsi-350000399b8420a0d +rootfs_disk: + - /dev/disk/by-id/scsi-350000399b841ff55 +mac_addr: + - 44:67:47:85:d4:9e + - 44:67:47:85:d4:9f + - 44:67:47:85:d4:a0 + - 44:67:47:85:d4:a1 + - 44:67:47:85:d4:a2 + - 44:67:47:85:d4:a3 + - 44:67:47:85:d4:a4 + - 44:67:47:85:d4:a5 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.5 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a60 b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a60 new file mode 100644 index 0000000..01aa6a6 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a60 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500bd67640b + - /dev/disk/by-id/scsi-35000c500bd6707a7 + - /dev/disk/by-id/scsi-35000c500bd680dc3 +rootfs_disk: + - /dev/disk/by-id/scsi-35000c500bd683a73 +mac_addr: + - e0:00:84:2b:51:7d + - e0:00:84:2b:51:7e + - e0:00:84:2b:51:7f + - e0:00:84:2b:51:80 + - e0:00:84:2b:51:81 + - e0:00:84:2b:51:82 + - e0:00:84:2b:51:83 + - e0:00:84:2b:51:84 +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.1 +queues: + taishan200-2280-2s48p-256g--multi-qemu diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a61 b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a61 new file mode 100644 index 0000000..e935de7 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a61 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a7869 + - /dev/disk/by-id/scsi-350000399886acc5d + - /dev/disk/by-id/scsi-350000399886b6861 +rootfs_disk: + - /dev/disk/by-id/scsi-350000399886ac70d +mac_addr: + - 44:67:47:c9:e4:b8 + - 44:67:47:c9:e4:b9 + - 44:67:47:c9:e4:ba + - 44:67:47:c9:e4:bb + - 44:67:47:c9:e4:bc + - 44:67:47:c9:e4:bd + - 44:67:47:c9:e4:be + - 44:67:47:c9:e4:bf +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.2 +queues: + taishan200-2280-2s48p-256g--multi-qemu diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a62 b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a62 new file mode 100644 index 0000000..03e043a --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/taishan200-2280-2s48p-256g--a62 @@ -0,0 +1,24 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b892d131 + - /dev/disk/by-id/scsi-350000399b892d619 + - /dev/disk/by-id/scsi-350000399b892dc1d +rootfs_disk: + - /dev/disk/by-id/scsi-350000399b892dbb5 +mac_addr: + - 44:67:47:c9:e3:3a + - 44:67:47:c9:e3:3b + - 44:67:47:c9:e3:3c + - 44:67:47:c9:e3:3d + - 44:67:47:c9:e3:3e + - 44:67:47:c9:e3:3f + - 44:67:47:c9:e3:40 + - 44:67:47:c9:e3:41 +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.3 +queues: + taishan200-2280-2s48p-256g--multi-qemu diff --git a/hosts/taishan200-2280-2s48p-256g--a11 b/hosts/taishan200-2280-2s48p-256g--a11 index 412154f..0a7f15e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a11 +++ b/hosts/taishan200-2280-2s48p-256g--a11 @@ -20,3 +20,5 @@ mac_addr: - cc:05:77:ab:97:69 model_name: Kunpeng-920 ipmi_ip: 9.3.5.1 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/hosts/taishan200-2280-2s48p-256g--a12 b/hosts/taishan200-2280-2s48p-256g--a12 index 3baa8f1..0ac46fe 100644 --- a/hosts/taishan200-2280-2s48p-256g--a12 +++ b/hosts/taishan200-2280-2s48p-256g--a12 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:97:22:3e model_name: Kunpeng-920 ipmi_ip: 9.3.5.2 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/hosts/taishan200-2280-2s48p-256g--a13 b/hosts/taishan200-2280-2s48p-256g--a13 index 4b9548f..5cc2506 100644 --- a/hosts/taishan200-2280-2s48p-256g--a13 +++ b/hosts/taishan200-2280-2s48p-256g--a13 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:97:2b:c6 model_name: Kunpeng-920 ipmi_ip: 9.3.5.3 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/hosts/taishan200-2280-2s48p-256g--a14 b/hosts/taishan200-2280-2s48p-256g--a14 index b96e79c..ce617ad 100644 --- a/hosts/taishan200-2280-2s48p-256g--a14 +++ b/hosts/taishan200-2280-2s48p-256g--a14 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:c9:e9:c9 model_name: Kunpeng-920 ipmi_ip: 9.3.5.4 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/hosts/taishan200-2280-2s48p-256g--a15 b/hosts/taishan200-2280-2s48p-256g--a15 index a1d63ed..ce2079e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a15 +++ b/hosts/taishan200-2280-2s48p-256g--a15 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:85:d4:a5 model_name: Kunpeng-920 ipmi_ip: 9.3.5.5 +queues: + taishan200-2280-2s48p-256g--multi-docker diff --git a/hosts/taishan200-2280-2s48p-256g--a60 b/hosts/taishan200-2280-2s48p-256g--a60 index 13612be..01aa6a6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a60 +++ b/hosts/taishan200-2280-2s48p-256g--a60 @@ -20,3 +20,5 @@ mac_addr: - e0:00:84:2b:51:84 model_name: Kunpeng-920 ipmi_ip: 9.3.4.1 +queues: + taishan200-2280-2s48p-256g--multi-qemu diff --git a/hosts/taishan200-2280-2s48p-256g--a61 b/hosts/taishan200-2280-2s48p-256g--a61 index 1619d17..e935de7 100644 --- a/hosts/taishan200-2280-2s48p-256g--a61 +++ b/hosts/taishan200-2280-2s48p-256g--a61 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:c9:e4:bf model_name: Kunpeng-920 ipmi_ip: 9.3.4.2 +queues: + taishan200-2280-2s48p-256g--multi-qemu diff --git a/hosts/taishan200-2280-2s48p-256g--a62 b/hosts/taishan200-2280-2s48p-256g--a62 index f1d72af..03e043a 100644 --- a/hosts/taishan200-2280-2s48p-256g--a62 +++ b/hosts/taishan200-2280-2s48p-256g--a62 @@ -20,3 +20,5 @@ mac_addr: - 44:67:47:c9:e3:41 model_name: Kunpeng-920 ipmi_ip: 9.3.4.3 +queues: + taishan200-2280-2s48p-256g--multi-qemu -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: optimize assign-account-fail-email
by Luan Shengde 28 Jan '21

28 Jan '21

add message for newly added error types. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 242dfe1..3597159 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -33,17 +33,20 @@ def email_err_message(message) https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… EMAIL_MESSAGE - when 'COMMIT_URL_NOT_AVAILABLE' + when 'COMMIT_AUTHOR_ERROR' err_message = <<~EMAIL_MESSAGE - We cannot confirm the commit url matches your email. + We cannot confirm the commit author matches your email. + Make sure it is truely submitted with your email. - Make sure the commit exists. - Make sure it is truely submitted with your email. - - Go to the commit history and find your commit. - Enter the commit and you will get your commit url. + Choose a commit that was submitted with your own email and try again. + EMAIL_MESSAGE + when 'NO_COMMIT_ID' + err_message = <<~EMAIL_MESSAGE + There is no such commit ID in the repo. - Try again after you have correctted the url + Please check the commit ID exists. + or + You have write the commit ID correctly. EMAIL_MESSAGE when 'NO_PUBKEY' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: fix bug for parsing email
by Luan Shengde 28 Jan '21

28 Jan '21

[why] for non-gitee repos: we may sometimes can not get the author email from the http request feedback when checking the commit. users may still receive a fail email even if they send 'apply account' email correctly. [how] both gitee and non-gitee will clone the repo tree and check the commit: - check commit id exists - check commit author matches the email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index a41a89a..95b967a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -6,7 +6,6 @@ require 'json' require 'mail' require 'rest-client' -require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -91,29 +90,34 @@ class ParseApplyAccountEmail end def commit_url_availability(url, base_url) - hub_name = url.split('/')[2] - - # it requires authentication when sending request to get the commit information - # clone the repo and then validate the commit for the email address - if hub_name.eql? 'gitee.com' - gitee_commit(url, base_url) - else - non_gitee_commit(url) - end - end + repo_url = [base_url, 'git'].join('.') + repo_dir = repo_url.split('/')[-1] + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) + + # get all commit IDs and check commit id exists + repo_commits = %x(git -C #{repo_dir} log --pretty=format:'%H').split(/\n/) + check_commit_exist(commit_id, repo_commits) - def gitee_commit(url, base_url) - my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) - my_gitee_commit.gitee_commit_check + # get the auther's email for the commit + author_email = %x(/usr/bin/git -C #{repo_dir} show -s --format=%aE #{commit_id}).chomp + check_commit_email(author_email) + + FileUtils.rm_rf repo_dir end - def non_gitee_commit(url) - url_fdback = RestClient.get(url).body - email_index = url_fdback.index @my_info['my_email'] + def check_commit_exist(commit_id, repo_commits) + return if repo_commits.include? commit_id + + raise 'NO_COMMIT_ID' + end - return unless email_index.nil? + def check_commit_email(author_email) + return if author_email.eql? @my_info['my_email'] - raise 'COMMIT_URL_NOT_AVAILABLE' + raise 'COMMIT_AUTHOR_ERROR' end def parse_pub_key -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 28 Jan '21

28 Jan '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 2 +- distro/archlinux | 2 +- distro/common | 12 ++++++++++++ distro/debian | 2 +- tests/cci-depends | 9 ++------- 5 files changed, 17 insertions(+), 10 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 598243d8e..5c234ada6 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -80,7 +80,7 @@ yum_download() yum_pack() { local target_dir=/opt/rpms - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_rpms=$(find /var/ -type f -name "*.rpm") diff --git a/distro/archlinux b/distro/archlinux index 3942080a6..3d576555c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -50,7 +50,7 @@ pacman_download() pacman_pack() { local target_dir=/opt/pkgs - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_pkgs=$(find_pacman_package) diff --git a/distro/common b/distro/common index 1fd7df4e8..0994dff30 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,15 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + [ -f ${BM_NAME}.cgz ] || return + + cd "$pack_to" + rm ${BM_NAME}.cgz + mv ${BM_NAME}_$date.cgz ${benchmark}_$date.cgz + + ln -sf ${benchmark}_$date.cgz ${benchmark}.cgz +} diff --git a/distro/debian b/distro/debian index 8821e3ddd..b07d395b6 100755 --- a/distro/debian +++ b/distro/debian @@ -99,7 +99,7 @@ fixup_perf() pack() { - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") [[ $(type -t fixup_$BM_NAME) = 'function' ]] && fixup_$BM_NAME diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 0

[PATCH lkp-tests 1/2] tests/cci-depends: delete inappropriate benchmark
by Wang Yong 28 Jan '21

28 Jan '21

[before] mount dest directory and pack each benchmark in loop when benchmark is all [after] now use curl to upload one file, it's inappropriate to pack all Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index c89737aca..8ac1d7e58 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -55,8 +55,6 @@ update fixup_preinstall -[[ "$benchmark" = "all" ]] && benchmark="$(ls $LKP_SRC/distro/depends | grep -v -e '-dev' -e '-allowlist')" - fixup_arch_install $pack_arch pack_benchmark_deps -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: export CCI_REPOS
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] LKP_SRC/lib/job.rb: lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if CCI_REPOS not exists, can't get the lab_hosts_file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 4638e15..46f450d 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -58,6 +58,7 @@ cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci export REPO_SRC=/c/upstream-repos +export CCI_REPOS=/c export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF -- 2.23.0

1 0

[PATCH v2 lab-z9] add host-file of a7 a20 a35
by Zhang Dewan 28 Jan '21

28 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a20 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a35 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a7 | 25 +++++++++++++++++++++++++ 3 files changed, 65 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a20 create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s64p-256g--a7 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 new file mode 100644 index 0000000..21bf8e1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b888f2dd + - /dev/disk/by-id/scsi-350000399b888f705 + - /dev/disk/by-id/scsi-350000399b88906f5 + - /dev/disk/by-id/scsi-350000399b8890141 +mac_addr: + - 84:46:fe:3d:a7:d5 + - 84:46:fe:3d:a7:d6 + - 84:46:fe:3d:a7:d7 + - 84:46:fe:3d:a7:d8 + - 84:46:fe:3d:a7:d9 + - 84:46:fe:3d:a7:da + - 84:46:fe:3d:a7:db + - 84:46:fe:3d:a7:dc +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.10 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..5d9d500 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,19 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 new file mode 100644 index 0000000..07701d3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039998134611 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 +mac_addr: + - 84:46:fe:89:be:20 + - 84:46:fe:89:be:21 + - 84:46:fe:89:be:22 + - 84:46:fe:89:be:23 + - ac:b3:b5:1e:d6:eb + - ac:b3:b5:1e:d6:ec + - ac:b3:b5:1e:d6:ed + - ac:b3:b5:1e:d6:ee + - ac:b3:b5:1e:d7:7b + - ac:b3:b5:1e:d7:7c + - ac:b3:b5:1e:d7:7d + - ac:b3:b5:1e:d7:7e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.7 -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: optimize borrow-machine.en.md
by Luan Shengde 28 Jan '21

28 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.en.md | 341 +++++++++----------------------- 1 file changed, 98 insertions(+), 243 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 6a1fe19..7e66f40 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -1,246 +1,101 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - -- Send an email to apply for an account. -- Receive an email from compass-ci-robot(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +# Prepare + +- Apply account +- Config default yaml files + +If you have not completed above works, reference to [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to finish it. + +# Apply testbox + +## 1. Generate local ssh pub key + + Use the command below to check the pub key exists: + + ls ~/.ssh/*.pub + + Generate one if you don't have one now: + + ssh-keygen + +## 2. Select the job yaml + + We have provided various examples for you in dir ~/lkp-tests/jobs: + + You can filtrate examples for borrowing machine with command below: + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. Submit job + + Command to submit jobs: + + for DCs: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + for VMs: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + for HWs: + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - You can view the logs in real time for the job with the command above。 + - You will directly login the testbox if the job runs successfully. + - And you will receive an email that contains login command and server configuration information. + - Only within the period of borrowing machine, you can access the textbox with the login command. + +## 4. Return testbox + + Return manually(recommended): + + Manually execute 'reboot' in time to return the testbox. + Avoid a waste of computer resource with no-load running. + + Return automatically on maturity: + + The testbox will be returned automatically if it has expired its service life. + + - All testboxes will be returned if it has been executed command 'reboot'. + - The testbox cannot be accessed any more after it has been returned. + - Apply a new one if you want continue to use it. # FAQ -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the runtime field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit-job.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. +## Customize the borrowing time + + Find key in the yaml file and edit its value according to your requirement. + + - The borrowing period can be calculated in days and hours. + - The maximum period is no more than 10 day. + +## Guidance for command submit + + See the usage and options for command 'submit' the command below: + + submit -h + + Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… + +## Available testbox + + For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **instruction:** + > + > - DCs: dc-xxx + > - VMs: vm-xxx + > - HWs: taishan200-2280-xxx + + >![](./../public_sys-resources/icon-notice.gif) **attention:** + > - It means that you choosed a specified physical machine if the testbox name is end with `--axx`. + > - You will need to wait if there are already tasks in the task queue for the machine. + > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. + +## Specify the OS + + About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… -- 2.23.0

1 0

[PATCH lkp-tests] add a host group in lkp-tests/hosts
by Zhang Dewan 28 Jan '21

28 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g | 1 + 1 file changed, 1 insertion(+) create mode 100644 hosts/taishan200-2280-2s48p-256g diff --git a/hosts/taishan200-2280-2s48p-256g b/hosts/taishan200-2280-2s48p-256g new file mode 100644 index 00000000..6399b995 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g @@ -0,0 +1 @@ +arch: aarch64 -- 2.23.0

2 2

[PATCH v3 compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 28 Jan '21

28 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 +++- container/os-cifs/start | 4 +++- container/os-nfs/start | 7 +++++-- container/result-cifs/start | 4 +++- 4 files changed, 14 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..119d1b1 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..47da260 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..8c08db8 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,12 +3,15 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + exit 1 } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + log_error "[ERROR] kernel module nfsd is missing (try: sudo modprobe nfsd)" + exit 1 } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..4f8c2b9 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm result-cifs -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] replace log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 ++-- container/os-cifs/start | 4 ++-- container/os-nfs/start | 4 ++-- container/result-cifs/start | 4 ++-- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 8 files changed, 15 insertions(+), 20 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..7577647 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..83c330c --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"%F %T $* [INFO]" +} + +log_debug() +{ + date +"%F %T $* [DEBUG]" +} + +log_warn() +{ + date +"%F %T $* [WARN]" >&2 +} + +log_error() +{ + date +"%F %T $* [ERROR]" >&2 +} -- 2.23.0

1 1

[PATCH compass-ci 2/2] replace log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 4 files changed, 7 insertions(+), 12 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: show the ssh connect with red background
by Cao Xueliang 28 Jan '21

28 Jan '21

The ssh connect information is very important, attract users' attention by the red background. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 4785477ec..9091c92a9 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -89,7 +89,7 @@ class Monitor web_socket.close cmd = "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" - puts cmd + puts "\033[41m#{cmd}\033[0m" cmd = "ssh-keygen -R #{ssh_host};" + cmd exec cmd -- 2.23.0

2 2

[PATCH v2 lkp-tests] jobs: add send email yaml
by Wei Jihui 28 Jan '21

28 Jan '21

[usage] submit job.yaml -i send-email.yaml send_email.subject=$subject submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=$subject values of subject are limited to: job_failed/job_debug/borrow_success/borrow_failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/send-email-on-fail.yaml | 7 +++++++ jobs/send-email.yaml | 6 ++++++ 2 files changed, 13 insertions(+) create mode 100644 jobs/send-email-on-fail.yaml create mode 100644 jobs/send-email.yaml diff --git a/jobs/send-email-on-fail.yaml b/jobs/send-email-on-fail.yaml new file mode 100644 index 00000000..a38eb52d --- /dev/null +++ b/jobs/send-email-on-fail.yaml @@ -0,0 +1,7 @@ +# use this send-email-on-fail.yaml by: +# submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=$subject +# values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. + +on_fail: + send_email: + subject: job_failed diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml new file mode 100644 index 00000000..8f8b48b9 --- /dev/null +++ b/jobs/send-email.yaml @@ -0,0 +1,6 @@ +# use this send_email.yaml by: +# submit job.yaml -i send-email.yaml send_email.subject=$subject +# values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. + +send_email: + subject: job_failed -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add send email yaml
by Wei Jihui 28 Jan '21

28 Jan '21

[usage] submit job.yaml -i send-email.yaml send_email.subject=xx submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=xx subject value limited at: job_failed/job_debug/borrow_success/borrow_failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/send-email-on-fail.yaml | 7 +++++++ jobs/send-email.yaml | 6 ++++++ 2 files changed, 13 insertions(+) create mode 100644 jobs/send-email-on-fail.yaml create mode 100644 jobs/send-email.yaml diff --git a/jobs/send-email-on-fail.yaml b/jobs/send-email-on-fail.yaml new file mode 100644 index 00000000..c81f756f --- /dev/null +++ b/jobs/send-email-on-fail.yaml @@ -0,0 +1,7 @@ +# use this job yaml by: +# submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=xx ... +# subject value limited at job_failed/job_debug/borrow_success/borrow_failed. + +on_fail: + send_email: + subject: job_failed diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml new file mode 100644 index 00000000..e6e767ba --- /dev/null +++ b/jobs/send-email.yaml @@ -0,0 +1,6 @@ +# use this job yaml by: +# submit job.yaml -i send-email.yaml send_email.subject=xx ... +# subject value limited at job_failed/job_debug/borrow_success/borrow_failed. + +send_email: + subject: job_failed -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/stats_wrapper.rb support handle $result_root/$script.yaml
by Lu Weitao 28 Jan '21

28 Jan '21

[Example] iperf.yaml: { tcp.sender.bps: 34804801216.197174 tcp.receiver.bps: 34804762215.18231 } iperf.json: { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 5bde4ab..c0406ad 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -24,7 +24,7 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" - return unless pretreatment + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile check_tmpfile @@ -180,6 +180,15 @@ module StatsWrapper system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") end + def self.create_yaml_tmpfile(file) + File.readlines(file).each do |line| + next if line =~ /{|}/ + + line.lstrip! + File.write(@tmpfile, line, mode: 'a') + end + end + def self.create_tmpfile tmp = Tempfile.new('lkp-stats.', '/tmp') @tmpfile = tmp.path @@ -191,6 +200,8 @@ module StatsWrapper check #{@tmpfile}") create_status = false end + elsif File.exist?("#{(a)log}.yaml") + create_yaml_tmpfile("#{(a)log}.yaml") else %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) unless $CHILD_STATUS.exitstatus.zero? -- 2.23.0

1 0

[PATCH compass-ci 3/3] lib/stats_wrapper.rb support handle $result_root/$script.yaml
by Lu Weitao 28 Jan '21

28 Jan '21

[Example] iperf.yaml: { tcp.sender.bps: 34804801216.197174 tcp.receiver.bps: 34804762215.18231 } iperf.json: { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 5bde4ab..40647c4 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -24,7 +24,7 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" - return unless pretreatment + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile check_tmpfile @@ -180,6 +180,13 @@ module StatsWrapper system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") end + def self.create_yaml_tmpfile(file) + File.readlines(file).each do |line| + next if line =~ %r{({|})} + line.lstrip! + File.write(@tmpfile, line, mode: 'a') + end + end def self.create_tmpfile tmp = Tempfile.new('lkp-stats.', '/tmp') @tmpfile = tmp.path @@ -191,6 +198,8 @@ module StatsWrapper check #{@tmpfile}") create_status = false end + elsif File.exist?("#{(a)log}.yaml") + create_yaml_tmpfile("#{(a)log}.yaml") else %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) unless $CHILD_STATUS.exitstatus.zero? -- 2.23.0

1 1

[PATCH compass-ci 2/3] translate lkp-tests/stats/wrapper to compass-ci/lib/stats_wrapper.rb
by Lu Weitao 28 Jan '21

28 Jan '21

translate lkp-tests/stats/wrapper(shell) to compass-ci/lib/stats_wrapper.rb(ruby) [Why] we need reduce calls between processes to improve performance Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 252 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 lib/stats_wrapper.rb diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb new file mode 100644 index 0000000..5bde4ab --- /dev/null +++ b/lib/stats_wrapper.rb @@ -0,0 +1,252 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# frozen_string_literal: true + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' +require "#{LKP_SRC}/lib/log.rb" +require 'tempfile' +require 'English' + +PROGRAM_DIR = "#{LKP_SRC}/stats" + +# exit processing the stats if the program is not in the program_list. +# program_list is a file that records all the programs(setups, monitors, tests, +# daemons) being executed during the test. +# this is to solve the problem for cluster test jobs running on the server node +# where no program's log file is generated for running server daemons but the +# stats processing for the program running on client node will also be handled +# on server node due to the current Job2sh algorithm. + +# default stats are not in the program_list +module StatsWrapper + def self.wrapper(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless pretreatment + return unless create_tmpfile + + check_tmpfile + warn_empty_stats + return unless create_stat + + check_empty_json + delete_log_package + end + + def self.pretreatment + return unless available_program? + return unless unzip_log + + extract_kmsg_dmesg + check_incomplete_run(@log) + check_soft_timeout + return if check_empty_output + return if check_binary_output + + true + end + + def self.available_program? + yaml_data = File.read("#{LKP_SRC}/etc/default_stats.yaml") + return true if yaml_data =~ /^#{@program}:/ + + return unless File.exist?("#{RESULT_ROOT}/program_list") + + pro_list = File.read("#{RESULT_ROOT}/program_list") + pro_list =~ /#{@program}/ + end + + def self.check_incomplete_run(file) + return if File.size?(file) + return unless File.exist?("#{LKP_SRC}/tests/#{@program}") + + data = "# missing #{@program} #{file}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_soft_timeout + return unless File.exist?("#{RESULT_ROOT}/soft_timeout") + + last_state = File.readlines("#{RESULT_ROOT}/last_state") + last_state.map!(&:chomp!) + data = 'soft_timeout: 1' + return if last_state.indlude?(data) + + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.delete_log + File.delete(@log) if File.exist?(@log) + File.delete("#{(a)log}.gz") if File.exist?("#{(a)log}.gz") + end + + def self.check_empty_output + return if File.size?(@log) + return if %w[tcrypt kernel-size perf-profile].include?(@program) + return if @program == 'dmesg' && File.size?("#{RESULT_ROOT}/kmsg") + return if @program == 'kmsg' && File.size?("#{RESULT_ROOT}/dmesg") + + delete_log + true + end + + def self.check_binary_output + return if %w[dmesg kmsg mpstat iostat].include?(@program) + + # kmsg may actually read the dmesg file + # refer to the exception cases in check_empty_output() + return unless File.exist?(@log) + return unless File.read(@log) =~ /\x0/ + + log_warn "skip binary file #{@log}" + + true + end + + def self.warn_empty_stats + return if File.size?(@tmpfile) + return if %w[dmesg ftrace turbostat perf-profile].include?(@program) + return unless File.size?("#{RESULT_ROOT}/time") + return if File.size?("#{RESULT_ROOT}/last_state") + return if File.read("#{LKP_SRC}/etc/failure") =~ /^#{(a)program}\./ + return if File.read("#{LKP_SRC}/etc/pass") =~ /^#{(a)program}\./ + + log_warn "empty stats for #{@log}" + end + + def self.check_exist_json + return if File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json") || File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json.gz") + return if File.exist?("#{RESULT_ROOT}/last_stat") && File.read("#{RESULT_ROOT}/last_state") =~ /is_incomplete_run/ + + log_warn "no generate json file for #{@stats_group}, check #{RESULT_ROOT}" + data = "# no json file for #{@stats_group}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_empty_json + testcase = RESULT_ROOT.split('/')[2] + testcase[4, testcase.length] if testcase =~ /^kvm:/ + + return if %w[borrow boot].include?(testcase) + return unless File.exist?("#{RESULT_ROOT}/job.sh") + + # testcase maybe different with run_case + # take fio-basic-1hdd-write.yaml job as example: + # - testcase will be fio-basc + # - run_case will be fio + + run_case = nil + job_data = File.readlines("#{RESULT_ROOT}/job.sh") + job_data.each do |line| + next unless line.chomp! =~ %r{LKP_SRC/tests/wrapper} + + parse = line.split(' ')[-1] + run_case = parse + break + end + + return unless run_case == @stats_group + + check_exist_json + end + + def self.unzip_log + if File.exist?("#{(a)log}.gz") + return unless File.size?("#{(a)log}.gz") + + system "zcat #{(a)log}.gz > #{@log}" + elsif File.exist?("#{(a)log}.xz") + return unless File.size?("#{(a)log}.xz") + + system "xzcat #{(a)log}.xz > #{@log}" + end + + true + end + + def self.extract_kmsg_dmesg + @kmsg_log = "#{RESULT_ROOT}/kmsg" + # extract kmsg for kmsg related stats + if @log =~ %r{^#{RESULT_ROOT}/(boot-memory|boot-time|tcrypt|dmesg)$} + system "xzcat #{(a)kmsg_log}.xz" if File.exist?("#{(a)kmsg_log}.xz") + end + + @dmesg_log = "#{RESULT_ROOT}/dmesg" + # extract dmesg for dmesg related stats + return unless @log =~ %r{^#{RESULT_ROOT}/kmsg$} + + system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") + end + + def self.create_tmpfile + tmp = Tempfile.new('lkp-stats.', '/tmp') + @tmpfile = tmp.path + create_status = true + if File.exist?(@log) + %x(#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} exit code #{$CHILD_STATUS.exitstatus}, + check #{@tmpfile}") + create_status = false + end + else + %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} < /dev/null exit code #{$CHILD_STATUS.exitstatus}, check #{@tmpfile}") + create_status = false + end + end + + create_status + end + + def self.check_tmpfile + str = File.read("#{LKP_SRC}/etc/failure") + str += File.read("#{LKP_SRC}/etc/pass") + + str =~ /^#{(a)program}\./ || check_incomplete_run(@tmpfile) + end + + def self.dump_stat(stats_group, file) + %x(#{LKP_SRC}/sbin/dump-stat #{stats_group} < #{file}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{LKP_SRC}/sbin/dump-stat #{@program} exit code #{$CHILD_STATUS.exitstatus}, check #{file}") + return nil + end + + File.delete(file) + + true + end + + def self.create_stat + if @program == 'ftrace' + Dir.foreach(RESULT_ROOT) do |file| + if file =~ /^#{@program}\..*\.yaml/ + stats_group = File.basename(file, '.yaml') + dump_stat(stats_group, file) + end + end + else + dump_stat(@stats_group, @tmpfile) + end + end + + def self.delete_log_package + if File.exist?("#{(a)log}.gz") + File.delete(@log) + elsif File.exist?("#{(a)log}.xz") + File.delete(@log) + end + + # delete temporarily extracted kmsg above + File.delete(@kmsg_log) if File.exist?("#{(a)kmsg_log}.xz") + + # delete temporarily extracted dmesg above + File.delete(@dmesg_log) if File.exist?("#{(a)dmesg_log}.xz") + + File.delete(@tmpfile) if File.exist?(@tmpfile) + end +end -- 2.23.0

1 0

[PATCH compass-ci 1/3] sbin/result2stats: get stats_program_list from job.yaml
by Lu Weitao 28 Jan '21

28 Jan '21

[Why] Get rid of dependence on job.sh [How] stats_program_list include: - stat-programs from job[key] which matched with lkp-tests/stats* - job[$suite] - time job[$suite].time - stderr - default stats: lkp-tests/etc/default_stats.yaml Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 +- lib/stats.rb | 63 +++++++++++++++++++++++++++++++++++++++++++++++ sbin/result2stats | 15 ++++------- 3 files changed, 69 insertions(+), 11 deletions(-) create mode 100644 lib/stats.rb diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 4dd03b4..526fadf 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -2,7 +2,7 @@ # frozen_string_literal: true -LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/stats" require "#{LKP_SRC}/lib/yaml" diff --git a/lib/stats.rb b/lib/stats.rb new file mode 100644 index 0000000..1ef6656 --- /dev/null +++ b/lib/stats.rb @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# frozen_string_literal: true + +require_relative './stats_wrapper' + +CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' + +# extract each result from the script output +# format each result as $script.json +class Stats + def initialize(result_root) + @result_root = result_root + @job = load_yaml(result_root + '/job.yaml') + end + + def extract_stats + ENV['RESULT_ROOT'] = @result_root + stats_list = assign_stats_list + stats_list.each do |stat| + StatsWrapper.wrapper(*stat) + end + end + + def assign_stats_list + stats_list = [] + @available_stats = available_stats + + stats_list << @job['suite'] + stats_list << ['time', @job['suite'] + '.time'] + stats_list << 'stderr' + @job.each_key do |k| + if @available_stats.include?(k) + stats_list << k + end + end + assign_default_stats(stats_list) + + stats_list + end +end + +def load_yaml(path) + unless File.file?(path) + warn "#{path} does not exist" + exit + end + YAML.load_file(path) +end + +def available_stats + vailable_stats = [] + Dir.open(LKP_SRC + '/stats').each do |file| + vailable_stats << file unless file.start_with?('.') + end + + vailable_stats +end + +def assign_default_stats(stats_list) + default_stats = YAML.load_file(LKP_SRC + '/etc/default_stats.yaml').keys + default_stats.each { |stat| stats_list << stat } +end diff --git a/sbin/result2stats b/sbin/result2stats index 6229b3c..fb7c18e 100755 --- a/sbin/result2stats +++ b/sbin/result2stats @@ -7,16 +7,10 @@ CCI_SRC = File.dirname(__dir__) require 'optparse' require "#{CCI_SRC}/lib/matrix2" +require_relative '../lib/stats.rb' File.umask 0o002 -def extract_stats(result_root) - ENV['RESULT_ROOT'] = result_root - job_script = result_root + '/job.sh' - - system job_script, 'extract_stats' -end - opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: result2stats result_root' opts.separator 'eg: result2stats /srv/result/iperf/vm-1p1g/2020-09-30/crystal.91198' @@ -36,7 +30,8 @@ argv = if ARGV == [] end opt_parser.parse!(argv) +RESULT_ROOT = argv[0] || ENV['RESULT_ROOT'] -result_root = argv[0] || ENV['RESULT_ROOT'] -extract_stats result_root -create_stats result_root +stats = Stats.new(RESULT_ROOT) +stats.extract_stats +create_stats RESULT_ROOT -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: add rsync module to docker image debian:dracut
by Xu Xijian 28 Jan '21

28 Jan '21

Local disk boot scheme need to add rsync module into initrd, docker image debian:dracut is used to make initrd, so add rsync to it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 2e676de..7e2bf51 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -6,7 +6,7 @@ export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y --no-install-recommends apt-utils >/dev/null 2>&1 -apt-get install -y nfs-common netbase cifs-utils kmod +apt-get install -y nfs-common netbase cifs-utils kmod rsync apt-get install -y dracut dracut-network dracut-config-generic apt-get clean -- 2.23.0

1 0

[PATCH v3 compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 28 Jan '21

28 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, lab, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..359a462 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,26 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

3 3

[PATCH v2 compass-ci 3/4] container: add start and register-accounts script for register-accounts container
by Cao Xueliang 28 Jan '21

28 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- .../register-accounts}/register-accounts | 1 + container/register-accounts/start | 35 +++++++++++++++++++ 2 files changed, 36 insertions(+) rename {sparrow/5-build => container/register-accounts}/register-accounts (98%) create mode 100755 container/register-accounts/start diff --git a/sparrow/5-build/register-accounts b/container/register-accounts/register-accounts similarity index 98% rename from sparrow/5-build/register-accounts rename to container/register-accounts/register-accounts index 41126f9..ff69032 100755 --- a/sparrow/5-build/register-accounts +++ b/container/register-accounts/register-accounts @@ -12,3 +12,4 @@ content.each do |key, value| build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) build_my_info.store_account_info end +sleep diff --git a/container/register-accounts/start b/container/register-accounts/start new file mode 100755 index 0000000..b1ebe98 --- /dev/null +++ b/container/register-accounts/start @@ -0,0 +1,35 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_CCI = '/c/compass-ci' +docker_rm 'register-accounts' + +cmd = %w[ + docker run + --name register-accounts + -d +] + env + %W[ + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/compass-ci/register:/etc/compass-ci/register:ro + -v /etc/localtime:/etc/localtime:ro + -w #{DEFAULT_CCI}/container/register-accounts + register-accounts +] + +cmd += ['sh', '-c', 'umask 002 && ./register-accounts'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci 1/4] sparrow: delete register-accounts in install-tiny
by Cao Xueliang 28 Jan '21

28 Jan '21

We will use a container to register accounts. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/install-tiny | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index eb9b3cb..a291549 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,5 +19,4 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe -5-build/register-accounts 7-systemd/systemd-setup -- 2.23.0

2 2

[PATCH lab-z9] modify the directory and files about idle
by Zhang Dewan 28 Jan '21

28 Jan '21

wo don't need so much multi-qemu, and reorganize the directory like this: dewan@crystal ~/tmp/lab-z9/allot/idle% tree . ├── multi-docker └── multi-qemu ├── taishan200-2280-2s48p-256g--a60 └── multi-qemu.yaml ├── taishan200-2280-2s48p-256g--a61 -> taishan200-2280-2s48p-256g--a60 ├── taishan200-2280-2s48p-256g--a62 -> taishan200-2280-2s48p-256g--a60 ├── taishan200-2280-2s48p-256g--a63 -> taishan200-2280-2s48p-256g--a60 └── taishan200-2280-2s48p-256g--a64 -> taishan200-2280-2s48p-256g--a60 7 directories, 1 file Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- .../taishan200-2280-2s48p-256g--a60/multi-qemu.yaml | 0 allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a61 | 0 allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a62 | 0 allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a63 | 0 allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a64 | 0 allot/idle/taishan200-2280-2s48p-256g--a11 | 1 - allot/idle/taishan200-2280-2s48p-256g--a12 | 1 - allot/idle/taishan200-2280-2s48p-256g--a13 | 1 - allot/idle/taishan200-2280-2s48p-256g--a14 | 1 - allot/idle/taishan200-2280-2s48p-256g--a15 | 1 - allot/idle/taishan200-2280-2s48p-256g--a16 | 1 - allot/idle/taishan200-2280-2s48p-256g--a17 | 1 - allot/idle/taishan200-2280-2s48p-256g--a18 | 1 - allot/idle/taishan200-2280-2s48p-256g--a19 | 1 - allot/idle/taishan200-2280-2s48p-256g--a20 | 1 - allot/idle/taishan200-2280-2s48p-256g--a21 | 1 - allot/idle/taishan200-2280-2s48p-256g--a22 | 1 - allot/idle/taishan200-2280-2s48p-256g--a24 | 1 - allot/idle/taishan200-2280-2s48p-256g--a25 | 1 - allot/idle/taishan200-2280-2s48p-256g--a26 | 1 - allot/idle/taishan200-2280-2s48p-256g--a27 | 1 - allot/idle/taishan200-2280-2s48p-256g--a28 | 1 - allot/idle/taishan200-2280-2s48p-256g--a29 | 1 - allot/idle/taishan200-2280-2s48p-256g--a30 | 1 - allot/idle/taishan200-2280-2s48p-256g--a31 | 1 - allot/idle/taishan200-2280-2s48p-256g--a32 | 1 - allot/idle/taishan200-2280-2s48p-256g--a33 | 1 - allot/idle/taishan200-2280-2s48p-256g--a34 | 1 - allot/idle/taishan200-2280-2s48p-256g--a35 | 1 - allot/idle/taishan200-2280-2s48p-256g--a65 | 1 - allot/idle/taishan200-2280-2s48p-256g--a66 | 1 - allot/idle/taishan200-2280-2s48p-256g--a67 | 1 - allot/idle/taishan200-2280-2s48p-256g--a68 | 1 - allot/idle/taishan200-2280-2s48p-256g--a69 | 1 - allot/idle/taishan200-2280-2s48p-256g--a70 | 1 - allot/idle/taishan200-2280-2s48p-256g--a71 | 1 - 36 files changed, 31 deletions(-) rename allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml (100%) rename allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a61 (100%) rename allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a62 (100%) rename allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a63 (100%) rename allot/idle/{ => multi-qemu}/taishan200-2280-2s48p-256g--a64 (100%) delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a11 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a12 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a13 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a14 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a15 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a16 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a17 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a18 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a19 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a20 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a21 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a22 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a24 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a25 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a26 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a27 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a28 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a29 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a30 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a31 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a32 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a33 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a34 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a35 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a71 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml rename to allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml diff --git a/allot/idle/taishan200-2280-2s48p-256g--a61 b/allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a61 similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a61 rename to allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a61 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a62 b/allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a62 similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a62 rename to allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a62 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a63 similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a63 rename to allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a63 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a64 similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a64 rename to allot/idle/multi-qemu/taishan200-2280-2s48p-256g--a64 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a11 b/allot/idle/taishan200-2280-2s48p-256g--a11 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a11 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a12 b/allot/idle/taishan200-2280-2s48p-256g--a12 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a12 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a13 b/allot/idle/taishan200-2280-2s48p-256g--a13 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a13 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a14 b/allot/idle/taishan200-2280-2s48p-256g--a14 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a14 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a15 b/allot/idle/taishan200-2280-2s48p-256g--a15 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a15 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a16 b/allot/idle/taishan200-2280-2s48p-256g--a16 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a16 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a17 b/allot/idle/taishan200-2280-2s48p-256g--a17 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a17 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a18 b/allot/idle/taishan200-2280-2s48p-256g--a18 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a18 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a19 b/allot/idle/taishan200-2280-2s48p-256g--a19 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a19 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a20 b/allot/idle/taishan200-2280-2s48p-256g--a20 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a20 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a21 b/allot/idle/taishan200-2280-2s48p-256g--a21 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a21 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a22 b/allot/idle/taishan200-2280-2s48p-256g--a22 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a22 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a24 b/allot/idle/taishan200-2280-2s48p-256g--a24 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a24 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a25 b/allot/idle/taishan200-2280-2s48p-256g--a25 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a25 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a26 b/allot/idle/taishan200-2280-2s48p-256g--a26 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a26 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a27 b/allot/idle/taishan200-2280-2s48p-256g--a27 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a27 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a28 b/allot/idle/taishan200-2280-2s48p-256g--a28 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a28 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a29 b/allot/idle/taishan200-2280-2s48p-256g--a29 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a29 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a30 b/allot/idle/taishan200-2280-2s48p-256g--a30 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a30 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a31 b/allot/idle/taishan200-2280-2s48p-256g--a31 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a31 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a32 b/allot/idle/taishan200-2280-2s48p-256g--a32 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a32 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a33 b/allot/idle/taishan200-2280-2s48p-256g--a33 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a33 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a34 b/allot/idle/taishan200-2280-2s48p-256g--a34 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a34 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a35 b/allot/idle/taishan200-2280-2s48p-256g--a35 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a35 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a65 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a66 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a67 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a68 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a69 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a70 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a71 b/allot/idle/taishan200-2280-2s48p-256g--a71 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a71 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file -- 2.23.0

4 6

[PATCH compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 28 Jan '21

28 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..9e0c671 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/warning/i) && !stat.match(/nr_warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/error|\.exit_code\./i) && !stat.match(/nr_error/i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] providers/libvirt: split common elements of domain.xml
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] some elements can be specifies by user. case 1: a complete domain.xml we should replace some elements of user's xml like os name ... case 2: default domain.xml generate a default domain.xml case 3: only specifies a cpu templates use user's cpu template replace default cpu template case 4: composition template user can specifies different tempaltes like cpu, disk, ... [how] split domain.xml then template combination based on user requirements Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 6 ++++++ providers/libvirt/templates/cpu.xml | 6 ++++++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 13 files changed, 60 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..7646c91 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,6 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> + diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..6991457 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,6 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +  +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..317866d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'><%= loader %></loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] providers/lib: introduce context.rb
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] providers/lib: introduce context.rb
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: set umask and selinux
by Liu Yinsi 28 Jan '21

28 Jan '21

[why] some container services need umask 002 to work normally, cci-network.service need disabled or permissive selinux, or will have permission denied questions. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..01c32b2 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -27,6 +27,14 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF +cat >> /etc/bashrc <<EOF +umask 002 +EOF + +source /etc/bashrc + +setenforce 0 + server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

2 1

[PATCH v2 compass-ci 4/4] container: add start-depends for register-accounts container
by Cao Xueliang 28 Jan '21

28 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/register-accounts/start-depends diff --git a/container/register-accounts/start-depends b/container/register-accounts/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/register-accounts/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container: add register-accounts container build and Dockerfile
by Cao Xueliang 28 Jan '21

28 Jan '21

This container will register some default accounts which will be used in others service. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/Dockerfile | 15 +++++++++++++++ container/register-accounts/build | 5 +++++ 2 files changed, 20 insertions(+) create mode 100644 container/register-accounts/Dockerfile create mode 100755 container/register-accounts/build diff --git a/container/register-accounts/Dockerfile b/container/register-accounts/Dockerfile new file mode 100644 index 0000000..71b7386 --- /dev/null +++ b/container/register-accounts/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport json yaml elasticsearch io-console diff --git a/container/register-accounts/build b/container/register-accounts/build new file mode 100755 index 0000000..e1dcfa4 --- /dev/null +++ b/container/register-accounts/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t register-accounts . -- 2.23.0

1 0

[PATCH compass-ci 3/4] container: add start and register-accounts script for register-accounts container
by Cao Xueliang 28 Jan '21

28 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- .../register-accounts}/register-accounts | 0 container/register-accounts/start | 36 +++++++++++++++++++ 2 files changed, 36 insertions(+) rename {sparrow/5-build => container/register-accounts}/register-accounts (100%) create mode 100755 container/register-accounts/start diff --git a/sparrow/5-build/register-accounts b/container/register-accounts/register-accounts similarity index 100% rename from sparrow/5-build/register-accounts rename to container/register-accounts/register-accounts diff --git a/container/register-accounts/start b/container/register-accounts/start new file mode 100755 index 0000000..36042b3 --- /dev/null +++ b/container/register-accounts/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_CCI = '/c/compass-ci' +docker_rm 'register-accounts' + +cmd = %w[ + docker run + --name register-accounts + -d + --rm +] + env + %W[ + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/compass-ci/register:/etc/compass-ci/register:ro + -v /etc/localtime:/etc/localtime:ro + -w #{DEFAULT_CCI}/container/register-accounts + register-accounts +] + +cmd += ['sh', '-c', 'umask 002 && ./register-accounts'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci] doc/manual: optimize borrow-machine.en.md
by Luan Shengde 28 Jan '21

28 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.en.md | 341 +++++++++----------------------- 1 file changed, 98 insertions(+), 243 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index 6a1fe19..921fa35 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -1,246 +1,101 @@ -# Prerequisites - -Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - -- Send an email to apply for an account. -- Receive an email from compass-ci-robot(a)qq.com. -- Configure the local environment. - -# Applying for a Test Machine (VM) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (VM). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (VM) after use. - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - -# Applying for a Test Machine (Physical Machine) - -1. Generate a local RSA private-public key pair. - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. Select the YAML file as required. - - A **lkp-tests** folder is stored in each user directory `/home/${USER}`. - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. Submit the YAML file and connect to the test machine (physical machine). - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 14:51:59 CST 2020 - - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - For more information about how to use the **submit** command, testbox options, and how to borrow the specified operating system, see the FAQ at the end of this document. - -4. Return the test machine (physical machine) after use. - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +# Prepare + +- Apply account +- Config default yaml files + +If have not completed above works, reference to [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to finish it. + +# Apply testbox + +## 1. Generate local ssh pub key + + Use the command below to check the pub key exists: + + ls ~/.ssh/*.pub + + Generate one if you don't have one now: + + ssh-keygen + +## 2. Select the job yaml + + We have provided various examples for you in dir ~/lkp-tests/jobs: + + You can filtrate examples for borrowing machine with command below: + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. Submit job + + Command to submit jobs: + + for DCs: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + for VMs: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + for HWs: + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - You can view the logs in real time for the job with the command above。 + - You will directly login the testbox if the job runs successfully. + - And you will receive an email that contains login command and server configuration information. + - Only within the period of borrowing machine, you can access the textbox with the login command. + +## 4. Return testbox + + Retuan manually(recommended): + + Manually execute 'reboot' in time to return the testbox. + Avoid a waste of computer resource with no-load running. + + Return automatically on maturity: + + The testbox will be returned automatically if it has expired its service life. + + - All testboxes will be returned if it has been executed command 'reboot'. + - The testbox cannot be accessed any more after it has been returned. + - Apply a new one if you want continue to use it. # FAQ -* How Do I Change the Duration of Keeping the Test Machine when Applying for It? - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the runtime field. - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # After changing the value, submit the request again. - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Guide to the **submit** Command - - Reference: [submit-job.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… - -* What Are the testbox Options? - - For details about the testbox options, visit https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts. - - > ![](./../public_sys-resources/icon-note.gif) **Note** - > - > VM testbox: vm-xxx - > - > PM testbox: taishan200-2280-xxx - - > ![](./../public_sys-resources/icon-notice.gif) **Notice** - > -> - If the testbox of a physical machine ends with `--axx`, a physical machine is specified. If a task is already in the task queue of the physical machine, the borrow task you submitted will not be processed until the previous task in the queue is completed. - > - If the testbox of a physical machine does not end with `-axx`, no physical machine is specified. In this case, the borrow task you submitted will be immediately allocated to idle physical machines in the cluster for execution. - -* How Do I Borrow the Specified Operating System? - - For details about the supported `os`, `os_arch`, and `os_version`, see [os-os\_verison-os\_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/…. +## Customize the borrowing time + + Find key in the yaml file and edit its value according to your requirement. + + - The borrowing period can be calculated in days and hours. + - The maximum period is no more than 10 day. + +## Guidance for command submit + + See the usage and options for command 'submit' the command below: + + submit -h + + Reference the following line to learn the advanced usage for command 'submit': + + [submit detailed usage](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/sub… + +## Available testbox + + For a full list of testbox, reference to https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **instruction:** + > + > - DCs: dc-xxx + > - VMs: vm-xxx + > - HWs: taishan200-2280-xxx + + >![](./../public_sys-resources/icon-notice.gif) **attention:** + > - It means that you choosed a specified physical machine if the testbox name is end with `--axx`. + > - You will need to wait if there are already tasks in the task queue for the machine. + > - Your job will be randomly assigned to a machine that meets the requirements if the testbox name is not end with '-axx'. + +## Specify the OS + + About supportted `os`, `os_arch`, `os_version`, reference to [os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/ma… -- 2.23.0

1 0