- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 26 ++++++++++++++++++++++ container/lifecycle/build | 18 ++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 36 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 82 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..ed5f6ee --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,26 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update + +RUN apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..5862403 --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -e CCI_SRC=#{DEFAULT_LKP} + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

3 2

[PATCH v2 lab-z9] add hosts-file of taishan200-2280-2s64p-256g--a5 which include NVMe SSD
by Zhang Dewan 25 Jan '21

25 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a5 | 25 +++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a6 | 25 +++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a5 create mode 100644 hosts/taishan200-2280-2s64p-256g--a6 diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 new file mode 100644 index 0000000..e6c11b3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981306c5 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 +mac_addr: + - 84:46:fe:26:d3:47 + - 84:46:fe:26:d3:48 + - 84:46:fe:26:d3:49 + - 84:46:fe:26:d3:4a + - ac:b3:b5:1e:b4:ab + - ac:b3:b5:1e:b4:ac + - ac:b3:b5:1e:b4:ad + - ac:b3:b5:1e:b4:ae + - ac:b3:b5:1e:ba:9b + - ac:b3:b5:1e:ba:9c + - ac:b3:b5:1e:ba:9d + - ac:b3:b5:1e:ba:9e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a6 b/hosts/taishan200-2280-2s64p-256g--a6 new file mode 100644 index 0000000..0944193 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a6 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981344d9 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000301 +mac_addr: + - 84:46:fe:89:ba:56 + - 84:46:fe:89:ba:57 + - 84:46:fe:89:ba:58 + - 84:46:fe:89:ba:59 + - ac:b3:b5:1e:b5:8b + - ac:b3:b5:1e:b5:8c + - ac:b3:b5:1e:b5:8d + - ac:b3:b5:1e:b5:8e + - ac:b3:b5:1e:b6:eb + - ac:b3:b5:1e:b6:ec + - ac:b3:b5:1e:b6:ed + - ac:b3:b5:1e:b6:ee +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.6 -- 2.23.0

1 0

[PATCH compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..d4d6ca1 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def performing_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..9a191ab --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find testboxs that are perforing tasks + # curl http://localhost:11311/performing_testbox?size=10&from=0 + get "/performing_testbox" do |env| + env.lifecycle.performing_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. [how] add SCHED_HOST SCHED_PORT on constants.rb Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ lib/sched_client.rb | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Jan '21

25 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs than runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..54affa1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/constants.rb: add SCHED_HOST SCHED_PORT field
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/compare_error_messages.rb: return true if job's field job_state is finished
by Lin Jiaxin 25 Jan '21

25 Jan '21

The job is considered to be credible when its' field job_state is finished. Return true early and no need to search for error filename. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 5f74be7..9c864a3 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -13,9 +13,12 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # the later_job_id is credible. def credible?(previous_job_id, later_job_id, error_id) es = ESQuery.new + later_es_result = es.query_by_id(later_job_id) + + return true if later_es_result['job_state'] == 'finished' previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') - later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') filenames_check = filenames_check(previous_result_file, later_result_file, error_id) -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: fix script hangs when no job now
by Liu Yinsi 25 Jan '21

25 Jan '21

[error] when no job now, $initrds is null, execute my-qemu.sh will hangs. lys@crystal ~/compass-ci/providers% ./my-qemu.sh hostname: vm-2p8g.lys-3917841 mac: 0a-4b-33-a0-96-69 DoneDone % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 79 100 79 0 0 19750 0 --:--:-- --:--:-- --:--:-- 19750 lys@crystal ~/compass-ci/providers/qemu% cat --help cat Copy standard input to standard output. when input is null, script will hang to waiting for input. [how] function set_options(): cat $initrds > $initrd before: execut cat command, then check after: check kernel, check initrds, if not null, execute cat command. in addition, check_option_value -> check_kernel, check_initrds, check_qemu convenient to call as needed. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 4d05964..b4e87fc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() done < $ipxe_script } -check_option_value() +check_kernel() { [ -n "$kernel" ] || { log_info "Can not find job for current hostname: $hostname." @@ -64,10 +64,16 @@ check_option_value() log_error "Can not find kernel file or kernel file is empty: $kernel." exit 1 } - +} + +check_qemu() +{ # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} +check_initrds() +{ [ -n "$initrds" ] || { log_error "The current initrds is null." exit 1 @@ -118,6 +124,8 @@ set_qemu() do command -v "$qemu" > /dev/null && break done + + check_qemu } print_message() @@ -209,8 +217,10 @@ write_logfile parse_ipxe_script +check_kernel +check_initrds + set_options -check_option_value print_message -- 2.23.0

2 2