- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4

[PATCH compass-ci 4/4] sparrow/5-build: pack lkp-tests
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for user locally deploy compass-ci, submit job will check lkp-$arch.cgz exist, so call container lkp-initrd/run in sparrow script, it is convenient and does not need users to operate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources index 48a740b..cea3b00 100755 --- a/sparrow/5-build/sources +++ b/sparrow/5-build/sources @@ -3,3 +3,5 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. $CCI_SRC/sbin/download-rootfs + +$CCI_SRC/container/lkp-initrd/run -- 2.23.0

2 2

[PATCH compass-ci] container/srv-http: bind mount volumn
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for locally deploy compass-ci, if their host can't access internet, we provider service for them to download docker images cgz file, then they no need to build images by themselves. usage: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/initrd/dockerimage/ima… Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 945cde0..fa56fca 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 01 Feb '21

01 Feb '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

2 1

[PATCH v2 compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 01 Feb '21

01 Feb '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

2 1

[PATCH v1 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..93bc9ce --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline : #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: check host files in $lab repos
by Li Ping 01 Feb '21

01 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..12f1e05 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + get_lab_hosts_file() set_defaults() end @@ -382,6 +383,22 @@ class Job end end + private def get_lab_hosts_file + if !@hash["host_file"]? + lab_hosts_file_name = @hash["testbox"] + error_msg = "hosts file not exist: #{lab_hosts_file_name}" + lab_hosts_file = "#{ENV["CCI_REPOS"]}/lab-#{@hash["lab"]}/hosts/#{lab_hosts_file_name}" + if File.file?(lab_hosts_file) + host_hash = YAML.parse(File.read(lab_hosts_file)) + @hash.merge!(JSON.parse(host_hash.to_json).as_h) + else + raise error_msg + end + else + @hash.delete("host_file") + end + end + private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete get_lab_host_file
by Li Ping 01 Feb '21

01 Feb '21

[why] lab_host_files are on the server side and should be read on the server side. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 867c3d193..087ca226e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -354,6 +354,7 @@ class Job if hosts_file hwconfig = load_yaml(hosts_file, {}) @job[source_file_symkey(hosts_file)] = nil + @job["host_file"] = "#{hosts_file}" @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig @load_hosts_done = true end @@ -369,26 +370,14 @@ class Job return @job.include?('tbox_group') end - def get_lab_hosts_file - lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" - - if File.file?(lab_hosts_file) - lab_hosts_file - else - nil - end - end - def get_hosts_file - lab_hosts_file = get_lab_hosts_file - return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" - return hosts_file if File.file?(hosts_file) - - raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" + if File.file?(hosts_file) + hosts_file + else + nil + end end def include_files -- 2.23.0

1 0

[PATCH v1 compass-ci] providers/libvirt: check host resource and download file
by Xiao Shenwei 01 Feb '21

01 Feb '21

if the host machine does not have the capability of executing tasks, raise error message. if not, download dependent file based on response Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index b4f8b4c..a500c31 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -8,6 +8,7 @@ require 'digest/md5' require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" +require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" def create_logger(hostname) filename = "#{hostname}.log" @@ -51,10 +52,13 @@ def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? + resource.parse_response(response) + context.merge!(resource.info) sched_client.delete_mac2host(mac) sched_client.delete_host2queues(hostname) end -- 2.23.0

1 0

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2