- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/2] sbin/install-dependencies: replace gem sources
by Liu Yinsi 01 Feb '21

01 Feb '21

Encapsulating functions install_gem_pkg for easy gem sources replacement Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index c16385e0f..012c45499 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -13,7 +13,6 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket git activesupport rest-client ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -21,7 +20,6 @@ linux_dep() else sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp fi - sudo gem install -f git activesupport rest-client faye-websocket ;; *) echo "$ID not support! please install dependencies manually." && exit 1 @@ -34,7 +32,12 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - sudo gem install -f git activesupport rest-client +} + +install_gem_pkg() +{ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + sudo gem install -f git activesupport rest-client faye-websocket } run() @@ -46,6 +49,8 @@ run() else echo "$DISTRO not supported!" && exit 1 fi + + install_gem_pkg } set_env() -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/install-dependencies: exit if conditions are not met
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] auto install dependencies must meet conditions, if not, exit script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index f2a9155f6..c16385e0f 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -24,7 +24,7 @@ linux_dep() sudo gem install -f git activesupport rest-client faye-websocket ;; *) - echo "$ID not support! please install dependencies manually." + echo "$ID not support! please install dependencies manually." && exit 1 ;; esac } @@ -44,7 +44,7 @@ run() elif is_system "Darwin"; then mac_dep else - echo "$DISTRO not supported!" + echo "$DISTRO not supported!" && exit 1 fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete fields that need to be reloaded for reproduction task
by Li Ping 01 Feb '21

01 Feb '21

[why] 1. load job from result's job yaml has key "my_email", like my_email: compass-ci-robot(a)qq.com my_name: compass-ci-robot lab: z9 2. load job from result's job yaml has key "SCHED_HOST" which from /etc/compass-ci/defaults/*.yaml, like SCHED_HOST: 172.168.131.2 SCHED_PORT: 3000 each of these fields needs to be reset when submit job.yaml for reprodution task. however, these fields already exist in yaml file, can not to be reload, so delete these fields when submit reproduction job. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 640b25a5f..867c3d193 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -258,6 +258,13 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + if hash.has_key?('id') + hash.delete_if { |key, _| key == "SCHED_HOST" } + hash.delete_if { |key, _| key == "SCHED_PORT" } + hash.delete_if { |key, _| key == "my_name" } + hash.delete_if { |key, _| key == "my_email" } + hash.delete_if { |key, _| key == "lab" } + end revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? -- 2.23.0

3 4

[PATCH v2 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..bcce612 --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline: #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4

[PATCH compass-ci 4/4] sparrow/5-build: pack lkp-tests
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for user locally deploy compass-ci, submit job will check lkp-$arch.cgz exist, so call container lkp-initrd/run in sparrow script, it is convenient and does not need users to operate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources index 48a740b..cea3b00 100755 --- a/sparrow/5-build/sources +++ b/sparrow/5-build/sources @@ -3,3 +3,5 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. $CCI_SRC/sbin/download-rootfs + +$CCI_SRC/container/lkp-initrd/run -- 2.23.0

2 2

[PATCH compass-ci] container/srv-http: bind mount volumn
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for locally deploy compass-ci, if their host can't access internet, we provider service for them to download docker images cgz file, then they no need to build images by themselves. usage: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/initrd/dockerimage/ima… Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 945cde0..fa56fca 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 01 Feb '21

01 Feb '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

2 1

[PATCH v2 compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 01 Feb '21

01 Feb '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

2 1

[PATCH v1 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..93bc9ce --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline : #{@info['cmdline']}") + end +end -- 2.23.0

2 2