- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..d4d6ca1 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def performing_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..9a191ab --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find testboxs that are perforing tasks + # curl http://localhost:11311/performing_testbox?size=10&from=0 + get "/performing_testbox" do |env| + env.lifecycle.performing_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. [how] add SCHED_HOST SCHED_PORT on constants.rb Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ lib/sched_client.rb | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Jan '21

25 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs than runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..54affa1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/constants.rb: add SCHED_HOST SCHED_PORT field
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/compare_error_messages.rb: return true if job's field job_state is finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' field job_state is finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..9c864a3 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -13,9 +13,12 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # the later_job_id is credible. def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + later_es_result = es.query_by_id(later_job_id) + + return true if later_es_result['job_state'] == 'finished' previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') - later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') filenames_check = filenames_check(previous_result_file, later_result_file, error_id) -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: fix script hangs when no job now
by Liu Yinsi 25 Jan '21

25 Jan '21

[error] when no job now, $initrds is null, execute my-qemu.sh will hangs. lys@crystal ~/compass-ci/providers% ./my-qemu.sh hostname: vm-2p8g.lys-3917841 mac: 0a-4b-33-a0-96-69 DoneDone % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 79 100 79 0 0 19750 0 --:--:-- --:--:-- --:--:-- 19750 lys@crystal ~/compass-ci/providers/qemu% cat --help cat Copy standard input to standard output. when input is null, script will hang to waiting for input. [how] function set_options(): cat $initrds > $initrd before: execut cat command, then check after: check kernel, check initrds, if not null, execute cat command. in addition, check_option_value -> check_kernel, check_initrds, check_qemu convenient to call as needed. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 4d05964..b4e87fc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() done < $ipxe_script } -check_option_value() +check_kernel() { [ -n "$kernel" ] || { log_info "Can not find job for current hostname: $hostname." @@ -64,10 +64,16 @@ check_option_value() log_error "Can not find kernel file or kernel file is empty: $kernel." exit 1 } - +} + +check_qemu() +{ # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} +check_initrds() +{ [ -n "$initrds" ] || { log_error "The current initrds is null." exit 1 @@ -118,6 +124,8 @@ set_qemu() do command -v "$qemu" > /dev/null && break done + + check_qemu } print_message() @@ -209,8 +217,10 @@ write_logfile parse_ipxe_script +check_kernel +check_initrds + set_options -check_option_value print_message -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/compare_error_messages.rb: return true if job state = finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' state = finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..9c864a3 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -13,9 +13,12 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # the later_job_id is credible. def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + later_es_result = es.query_by_id(later_job_id) + + return true if later_es_result['job_state'] == 'finished' previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') - later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') filenames_check = filenames_check(previous_result_file, later_result_file, error_id) -- 2.23.0

2 2

[PATCH v1 compass-ci] lib: introduce log.rb
by Xiao Shenwei 25 Jan '21

25 Jan '21

Records the run logs of the ruby program message format: 2021-01-25 11:00:09 +0800 INFO -- SCHED_PORT: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/log.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..762c0b2 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# records the run logs of the ruby program +class Log < Logger + def initialize(filename) + File.delete(filename) if FileTest.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + transform_msg(msg).map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end + + def transform_msg(msg) + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + return msg + end +end -- 2.23.0

2 3

2025

2024

2023

2022

2021

2020

Compass-ci