- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/srv-http: bind mount volumn
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for locally deploy compass-ci, if their host can't access internet, we provider service for them to download docker images cgz file, then they no need to build images by themselves. usage: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/initrd/dockerimage/ima… Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 945cde0..fa56fca 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 01 Feb '21

01 Feb '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

2 1

[PATCH v2 compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 01 Feb '21

01 Feb '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

2 1

[PATCH v1 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..93bc9ce --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline : #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: check host files in $lab repos
by Li Ping 01 Feb '21

01 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..12f1e05 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + get_lab_hosts_file() set_defaults() end @@ -382,6 +383,22 @@ class Job end end + private def get_lab_hosts_file + if !@hash["host_file"]? + lab_hosts_file_name = @hash["testbox"] + error_msg = "hosts file not exist: #{lab_hosts_file_name}" + lab_hosts_file = "#{ENV["CCI_REPOS"]}/lab-#{@hash["lab"]}/hosts/#{lab_hosts_file_name}" + if File.file?(lab_hosts_file) + host_hash = YAML.parse(File.read(lab_hosts_file)) + @hash.merge!(JSON.parse(host_hash.to_json).as_h) + else + raise error_msg + end + else + @hash.delete("host_file") + end + end + private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete get_lab_host_file
by Li Ping 01 Feb '21

01 Feb '21

[why] lab_host_files are on the server side and should be read on the server side. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 867c3d193..087ca226e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -354,6 +354,7 @@ class Job if hosts_file hwconfig = load_yaml(hosts_file, {}) @job[source_file_symkey(hosts_file)] = nil + @job["host_file"] = "#{hosts_file}" @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig @load_hosts_done = true end @@ -369,26 +370,14 @@ class Job return @job.include?('tbox_group') end - def get_lab_hosts_file - lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" - - if File.file?(lab_hosts_file) - lab_hosts_file - else - nil - end - end - def get_hosts_file - lab_hosts_file = get_lab_hosts_file - return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" - return hosts_file if File.file?(hosts_file) - - raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" + if File.file?(hosts_file) + hosts_file + else + nil + end end def include_files -- 2.23.0

1 0

[PATCH v1 compass-ci] providers/libvirt: check host resource and download file
by Xiao Shenwei 01 Feb '21

01 Feb '21

if the host machine does not have the capability of executing tasks, raise error message. if not, download dependent file based on response Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index b4f8b4c..a500c31 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -8,6 +8,7 @@ require 'digest/md5' require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" +require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" def create_logger(hostname) filename = "#{hostname}.log" @@ -51,10 +52,13 @@ def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? + resource.parse_response(response) + context.merge!(resource.info) sched_client.delete_mac2host(mac) sched_client.delete_host2queues(hostname) end -- 2.23.0

1 0

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2

[PATCH v5 compass-ci 2/2] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] providers/lib: introduce context.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] expand erb templates need a class to save context [how] Context class provides a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci