- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 3/5] upload-libvirt-xml/start: start upload-libvirt-xml docker container
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/upload-libvirt-xml/start diff --git a/container/upload-libvirt-xml/start b/container/upload-libvirt-xml/start new file mode 100755 index 0000000..ddb04e7 --- /dev/null +++ b/container/upload-libvirt-xml/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +: ${UPLOAD_LIBVIRT_PORT:=3081} + +docker_rm upload-libvirt-xml + +cmd=( + docker run -d + --name upload-libvirt-xml + --restart=always + -p ${UPLOAD_LIBVIRT_PORT}:3081 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml + upload-libvirt-xml +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lib/sched_client.rb: add some methods for provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] we have some providers: ipxe, container, libvirt all of them need register testbox info and consume job from scheduler Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/sched_client.rb | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/lib/sched_client.rb b/lib/sched_client.rb index fec4b14..db60fa2 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -16,4 +16,43 @@ class SchedClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/submit_job") resource.post(job_json) end + + # scheduler API /boot.:boot_type/:parameter/:value + # boot_type parameter + # 1 ipxe mac + # 2 container hostname + # 3 libvirt mac + def consume_job(boot_type: '', parameter: '', value: '') + RestClient.get "http://#{@host}:#{@port}/boot.#{boot_type}/#{parameter}/#{value}" + end + + # some provider only need register host2queues + # like boot.container + def register_mac2host(hostname, mac) + @hostname = hostname + @mac = mac + RestClient.put( + "http://#{@host}:#{@port}/set_host_mac?hostname=#{@hostname}&mac=#{@mac}", {} + ) + end + + def register_host2queues(hostname, queues) + @hostname = hostname + @queues = queues + RestClient.put( + "http://#{@host}:#{@port}/set_host2queues?host=#{@hostname}&queues=#{@queues}", {} + ) + end + + def delete_mac2host + RestClient.put( + "http://#{@host}:#{@port}/del_host_mac?mac=#{@mac}", {} + ) + end + + def delete_host2queues + RestClient.put( + "http://#{@host}:#{@port}/del_host2queues?host=#{@hostname}", {} + ) + end end -- 2.23.0

2 2

[PATCH lkp-tests] lib/bootstrap: delete extra space
by Liu Yinsi 26 Jan '21

26 Jan '21

before: PUB_IP=$( ip route ... after: PUB_IP=$(ip route ... Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..1127de140 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -287,7 +287,7 @@ export_ip_mac() { if has_cmd ip; then export PUB_NIC=$(ip route get 1.2.3.4 | awk '{print $5; exit}') - export PUB_IP=$( ip route get 1.2.3.4 | awk '{print $7; exit}') + export PUB_IP=$(ip route get 1.2.3.4 | awk '{print $7; exit}') elif has_cmd route; then export PUB_NIC=$(route -n | awk '/[UG][UG]/ {print $8}') export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') -- 2.23.0

1 0

[PATCH v3 compass-ci 4/5] upload-libvirt-xml/openresty.sh: start openresty in upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/openresty.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/upload-libvirt-xml/openresty.sh diff --git a/container/upload-libvirt-xml/openresty.sh b/container/upload-libvirt-xml/openresty.sh new file mode 100755 index 0000000..6215d91 --- /dev/null +++ b/container/upload-libvirt-xml/openresty.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +umask 002 +/usr/local/openresty/bin/openresty -g 'daemon off;' -- 2.23.0

1 0

[PATCH v3 compass-ci 2/5] upload-libvirt-xml/Dockerfile: add upload-libvirt-xml docker image
by cuiyili 26 Jan '21

26 Jan '21

Add upload-libvirt-xml docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/upload-libvirt-xml/Dockerfile diff --git a/container/upload-libvirt-xml/Dockerfile b/container/upload-libvirt-xml/Dockerfile new file mode 100644 index 0000000..db48bcb --- /dev/null +++ b/container/upload-libvirt-xml/Dockerfile @@ -0,0 +1,14 @@ +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM openresty/openresty:alpine + +RUN apk add bash +RUN mkdir -p /var/log/nginx/ +RUN adduser lkp -u 1090 -g 1090 -D + +COPY ./openresty.sh /usr/local/bin/ + +CMD ["/usr/local/bin/openresty.sh"] -- 2.23.0

1 0

[PATCH compass-ci 2/4] sparrow/3-code: set default values
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] $CCI_SRC/sbin/download-rootfs: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz download rootfs needs SRV_HTTP_PORT, so set default values. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 2ba9662..4638e15 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -22,6 +22,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${RESULT_WEBDAV_PORT:=3080} : ${ES_HOST:=$server_ip} : ${ES_PORT:=9200} +: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -50,6 +51,7 @@ RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT lab: nolab ES_HOST: $ES_HOST ES_PORT: $ES_PORT +SRV_HTTP_PORT: $SRV_HTTP_PORT EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 26 Jan '21

26 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..a577b3e --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 26 Jan '21

26 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 +++++++++++++++++++++++ container/lifecycle/build | 18 +++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 33 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 78 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..d89a181 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..7e67b3c --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

3 2

[PATCH compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 26 Jan '21

26 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 3 ++- container/os-cifs/start | 3 ++- container/os-nfs/start | 6 ++++-- container/result-cifs/start | 3 ++- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..c70b318 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..6df167f 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..3b0ebb5 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -5,10 +5,12 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + echo "[ERROR] nfs module is not loaded in the Docker host's kernel (try: sudo modprobe nfs)" + exit } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + echo "[ERROR] nfsd module is not loaded in the Docker host's kernel (try: sudo modprobe nfsd)" + exit } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..3992cdd 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -5,7 +5,8 @@ . $CCI_SRC/container/defconfig.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + echo "[ERROR] cifs module is not loaded in the Docker host's kernel (try: sudo modprobe cifs)" + exit } docker_rm result-cifs -- 2.23.0

2 2

[PATCH v2 compass-ci] lib: introduce log.rb
by Xiao Shenwei 26 Jan '21

26 Jan '21

Records the run logs of the ruby program message format: 2021-01-25 11:00:09 +0800 INFO -- SCHED_PORT: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/log.rb | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..25c92c6 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# records the run logs of the ruby program +class Log < Logger + def initialize(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + transform_msg(msg).map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end + + def transform_msg(msg) + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + return msg + end +end -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends: fix install sysbench pkg depends
by Wei Jihui 26 Jan '21

26 Jan '21

[error] Error: Unable to find a match: libmysqlclient-dev [why] openeuler has not libmysqlclient-dev: mysql-devel change it to libmariadbclient-dev Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/adaptation/openeuler | 1 - distro/depends/sysbench-dev | 2 +- distro/depends/sysbench-mysql-dev | 1 - 3 files changed, 1 insertion(+), 3 deletions(-) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 1938a8aa..9fd38b66 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -281,7 +281,6 @@ dh-python: dh-strip-nondeterminism: dh-systemd: guile-2.0: guile -libmysqlclient-dev: mysql-devel libipc-run-perl: libklibc-dev: libpython2.7: diff --git a/distro/depends/sysbench-dev b/distro/depends/sysbench-dev index 3fa67deb..f570c463 100644 --- a/distro/depends/sysbench-dev +++ b/distro/depends/sysbench-dev @@ -1,2 +1,2 @@ pkg-config -libmysqlclient-dev +libmariadbclient-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index ca5b03d8..65633b33 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,3 +1,2 @@ pkg-config -libmysqlclient-dev mysql-server -- 2.23.0

2 1

[PATCH compass-ci] container/*: replace gem sources
by Lin Jiaxin 26 Jan '21

26 Jan '21

gr "http://rubygems.org" | awk -F':' '{print $1}' | xargs sed -i 's|echo '\'':sources: \["http://rubygems.org"\]'\'' >> ~/.gemrc|gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/|g' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/assist-result/Dockerfile | 2 +- container/auto-submit/Dockerfile | 2 +- container/extract-stats/Dockerfile | 2 +- container/fluentd-base/Dockerfile | 2 +- container/git-mirror/Dockerfile | 2 +- container/mail-robot/Dockerfile | 2 +- container/monitoring/Dockerfile | 2 +- container/scheduler-dev/Dockerfile | 2 +- container/scheduler/Dockerfile | 2 +- container/webhook/Dockerfile | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 60a2fbf..1a02f49 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -11,5 +11,5 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport json yaml elasticsearch sinatra puma terminal-table diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5804642..2a64c61 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -11,7 +11,7 @@ RUN apk update && \ apk add --no-cache git && \ apk add ruby-dev make gcc g++ util-linux -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN umask 002 && \ gem install bunny json activesupport git rest-client io-console elasticsearch diff --git a/container/extract-stats/Dockerfile b/container/extract-stats/Dockerfile index 31f4ef0..3a8c8f3 100644 --- a/container/extract-stats/Dockerfile +++ b/container/extract-stats/Dockerfile @@ -9,7 +9,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'grep' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport git json yaml elasticsearch ENV RUNTIME_DIR /c/cci/extract diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index ef671c2..08bcd9c 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -13,7 +13,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install fluentd && \ gem install fluent-plugin-rabbitmq && \ gem install fluent-plugin-elasticsearch && \ diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 8b2a50a..2eac95c 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,7 +10,7 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ apk add ruby-dev make gcc g++ openssh diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index f0aebbd..4402543 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -11,5 +11,5 @@ RUN apt-get update && \ apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install fileutils elasticsearch activesupport diff --git a/container/monitoring/Dockerfile b/container/monitoring/Dockerfile index 5ceaf4f..0a5dc22 100644 --- a/container/monitoring/Dockerfile +++ b/container/monitoring/Dockerfile @@ -8,7 +8,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client && \ gem install activesupport && \ gem install git && \ diff --git a/container/scheduler-dev/Dockerfile b/container/scheduler-dev/Dockerfile index e0c9b4a..3b68d4a 100644 --- a/container/scheduler-dev/Dockerfile +++ b/container/scheduler-dev/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER chief <taxcom(a)tom.com> RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport bigdecimal open3 && \ gem install git json yaml && \ gem install cucumber diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index edac026..0ad813c 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -11,7 +11,7 @@ RUN apk add --no-cache 'ruby-dev' \ 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client && \ gem install activesupport && \ gem install git && \ diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile index 9299bb6..1d5ee7c 100644 --- a/container/webhook/Dockerfile +++ b/container/webhook/Dockerfile @@ -13,7 +13,7 @@ RUN apk update && \ apk add --no-cache git && \ apk add npm ruby-dev -RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN npm install git-webhook-handler && \ gem install bunny -- 2.23.0

2 1

[PATCH compass-ci] sbin/es-jobs: uniform wrap style
by Lin Jiaxin 26 Jan '21

26 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/es-jobs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/es-jobs b/sbin/es-jobs index c055fe9..ff0d540 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -5,7 +5,6 @@ require 'optparse' require 'json' - require_relative '../lib/es_jobs' def parse_argv @@ -19,6 +18,7 @@ def parse_argv end items end + opt_refine = [-1] opt_fields = [] opt_stats_filter = [] -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/compare_matrixes.rb: fix TypeError
by Lu Weitao 25 Jan '21

25 Jan '21

[Why] When use compare as: sbin/compare id=crystal.120202 id=crystal.122057 met error: /home/lwt/work4/compass-ci/lib/compare_matrixes.rb:182:in `[]': no implicit conversion of String into Integer (TypeError) root cause: we call the function: def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) like: compare_matrixes(matrices_list, suite_list, options: options) the "options" will passed as "latest_jobs" [How] set nil as default value to "latest_jobs" like: def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..12c1f3e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -179,6 +179,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) end def latest_failure?(field, latest_jobs) + return true unless latest_jobs latest_jobs.any? { |job| job['stats'][field] } end @@ -203,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_matrixes.rb: fix TypeError
by Lu Weitao 25 Jan '21

25 Jan '21

[Why] When use compare as: sbin/compare id=crystal.120202 id=crystal.122057 met error: /home/lwt/work4/compass-ci/lib/compare_matrixes.rb:182:in `[]': no implicit conversion of String into Integer (TypeError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..12c1f3e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -179,6 +179,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) end def latest_failure?(field, latest_jobs) + return true unless latest_jobs latest_jobs.any? { |job| job['stats'][field] } end @@ -203,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 25 ++++++++++++++++++++++ container/lifecycle/build | 18 ++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 35 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 80 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..d89a181 --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,25 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..35d4198 --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,35 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +#DEFAULT_CCI = '/c/cci' +#DEFAULT_LKP = '/c/lkp-tests' +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..a577b3e --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def get_running_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..4154a27 --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find the testbox that are performing jobs + # curl http://localhost:11311/get_running_testbox?size=10&from=0 + get "/get_running_testbox" do |env| + env.lifecycle.get_running_testbox.to_json + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/lifecycle: add a new docker service
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/lifecycle/Dockerfile | 26 ++++++++++++++++++++++ container/lifecycle/build | 18 ++++++++++++++++ container/lifecycle/build-depends | 1 + container/lifecycle/start | 36 +++++++++++++++++++++++++++++++ container/lifecycle/start-depends | 1 + 5 files changed, 82 insertions(+) create mode 100644 container/lifecycle/Dockerfile create mode 100755 container/lifecycle/build create mode 100755 container/lifecycle/build-depends create mode 100755 container/lifecycle/start create mode 100755 container/lifecycle/start-depends diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile new file mode 100644 index 0000000..ed5f6ee --- /dev/null +++ b/container/lifecycle/Dockerfile @@ -0,0 +1,26 @@ +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update + +RUN apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client && \ + gem install activesupport && \ + gem install git && \ + gem install json && \ + gem install yaml + + +ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle + +RUN mkdir -p $LIFECYCLE_RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $LIFECYCLE_RUNTIME_DIR + +COPY --chown=1090:1090 lifecycle . diff --git a/container/lifecycle/build b/container/lifecycle/build new file mode 100755 index 0000000..8c4acca --- /dev/null +++ b/container/lifecycle/build @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new ['LIFECYCLE_PORT'] +defaults = relevant_defaults(names) + +LIFECYCLE_PORT = (defaults['LIFECYCLE_PORT'] || '11311') + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." + +system 'rm lifecycle' diff --git a/container/lifecycle/build-depends b/container/lifecycle/build-depends new file mode 100755 index 0000000..5dd403e --- /dev/null +++ b/container/lifecycle/build-depends @@ -0,0 +1 @@ +scheduler-dev diff --git a/container/lifecycle/start b/container/lifecycle/start new file mode 100755 index 0000000..5862403 --- /dev/null +++ b/container/lifecycle/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIFECYCLE_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' +LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11311' +docker_rm "lifecycle-#{LIFECYCLE_PORT}" + +cmd = %W[ + docker run + --restart=always + --name lifecycle-#{LIFECYCLE_PORT} + -d + -u 1090:1090 + -p #{LIFECYCLE_PORT}:#{LIFECYCLE_PORT} +] + env + %W[ + -e CCI_SRC=#{DEFAULT_LKP} + -v /etc/localtime:/etc/localtime:ro + lifecycle-#{LIFECYCLE_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./lifecycle'] + +system(*cmd) diff --git a/container/lifecycle/start-depends b/container/lifecycle/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/lifecycle/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

3 2

[PATCH v2 lab-z9] add hosts-file of taishan200-2280-2s64p-256g--a5 which include NVMe SSD
by Zhang Dewan 25 Jan '21

25 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a5 | 25 +++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a6 | 25 +++++++++++++++++++++++++ 2 files changed, 50 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a5 create mode 100644 hosts/taishan200-2280-2s64p-256g--a6 diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 new file mode 100644 index 0000000..e6c11b3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981306c5 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 +mac_addr: + - 84:46:fe:26:d3:47 + - 84:46:fe:26:d3:48 + - 84:46:fe:26:d3:49 + - 84:46:fe:26:d3:4a + - ac:b3:b5:1e:b4:ab + - ac:b3:b5:1e:b4:ac + - ac:b3:b5:1e:b4:ad + - ac:b3:b5:1e:b4:ae + - ac:b3:b5:1e:ba:9b + - ac:b3:b5:1e:ba:9c + - ac:b3:b5:1e:ba:9d + - ac:b3:b5:1e:ba:9e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a6 b/hosts/taishan200-2280-2s64p-256g--a6 new file mode 100644 index 0000000..0944193 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a6 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981344d9 +ssd_partitions: + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000301 +mac_addr: + - 84:46:fe:89:ba:56 + - 84:46:fe:89:ba:57 + - 84:46:fe:89:ba:58 + - 84:46:fe:89:ba:59 + - ac:b3:b5:1e:b5:8b + - ac:b3:b5:1e:b5:8c + - ac:b3:b5:1e:b5:8d + - ac:b3:b5:1e:b5:8e + - ac:b3:b5:1e:b6:eb + - ac:b3:b5:1e:b6:ec + - ac:b3:b5:1e:b6:ed + - ac:b3:b5:1e:b6:ee +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.6 -- 2.23.0

1 0

[PATCH compass-ci 2/3] service/lifecycle: add a new service
by Wu Zhende 25 Jan '21

25 Jan '21

Managing the life cycles of jobs and testboxs. Includes a series of query interfaces and data maintenance interfaces. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 39 ++++++++++++++++++++++++++++++++++++++ src/lib/web_env.cr | 9 +++++++++ src/lifecycle.cr | 16 ++++++++++++++++ src/lifecycle/constants.cr | 4 ++++ src/lifecycle/lifecycle.cr | 32 +++++++++++++++++++++++++++++++ 5 files changed, 100 insertions(+) create mode 100644 src/lib/lifecycle.cr create mode 100644 src/lifecycle.cr create mode 100644 src/lifecycle/constants.cr create mode 100644 src/lifecycle/lifecycle.cr diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr new file mode 100644 index 0000000..d4d6ca1 --- /dev/null +++ b/src/lib/lifecycle.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" +require "yaml" + +require "./web_env" +require "../scheduler/elasticsearch_client" + +class Lifecycle + property es + + def initialize(env : HTTP::Server::Context) + @es = Elasticsearch::Client.new + @env = env + @log = env.log.as(JSONLogger) + end + + def alive(version) + "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) + end + + def performing_testbox + size = @env.params.query["size"]? || 20 + from = @env.params.query["from"]? || 0 + query = { + "size" => size, + "from" => from, + "query" => { + "terms" => { + "state" => ["booting", "running"] + } + } + } + @es.search("testbox", query) + end +end diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index d33b4c0..b3a90a8 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "./sched" +require "./lifecycle" require "./json_logger" class HTTP::Server @@ -22,5 +23,13 @@ class HTTP::Server def log @log ||= create_log end + + def lifecycle + @lifecycle ||= create_lifecycle + end + + def create_lifecycle + @lifecycle = Lifecycle.new(self) + end end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr new file mode 100644 index 0000000..a864621 --- /dev/null +++ b/src/lifecycle.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "lifecycle/lifecycle" +require "./lifecycle/constants.cr" +require "./lib/json_logger" + +module Cycle + log = JSONLogger.new + + begin + Kemal.run(LIFECYCLE_PORT) + rescue e + log.error(e) + end +end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr new file mode 100644 index 0000000..a428251 --- /dev/null +++ b/src/lifecycle/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11311).to_i32 diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr new file mode 100644 index 0000000..9a191ab --- /dev/null +++ b/src/lifecycle/lifecycle.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/lifecycle" +require "../lib/json_logger" + +module Cycle + VERSION = "0.1.0" + + add_context_storage_type(Time::Span) + + before_all do |env| + env.set "start_time", Time.monotonic + env.response.headers["Connection"] = "close" + env.create_log + env.create_lifecycle + end + + # echo alive + get "/" do |env| + env.lifecycle.alive(VERSION) + end + + # find testboxs that are perforing tasks + # curl http://localhost:11311/performing_testbox?size=10&from=0 + get "/performing_testbox" do |env| + env.lifecycle.performing_testbox.to_json + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] scheduler/elasticsearch_client.cr: add a new function for searching
by Wu Zhende 25 Jan '21

25 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index c1663c9..e343809 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -78,6 +78,10 @@ class Elasticsearch::Client @client.get_source(query) end + def search(index, query) + @client.search({:index => index, :body => query}) + end + def update_account(account_content : JSON::Any, my_email : String) return @client.update( { -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/sched_client.rb: obtain host port from the configuration file
by Xiao Shenwei 25 Jan '21

25 Jan '21

[why] lib/sched_client.rb need host port of scheduler, obtain these values from the configuration file. [how] add SCHED_HOST SCHED_PORT on constants.rb Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/constants.rb | 3 +++ lib/sched_client.rb | 5 ++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index ffc4f03..27f37cb 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -18,3 +18,6 @@ SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.co ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 + +SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = config['SCHED_PORT'] || 3000 diff --git a/lib/sched_client.rb b/lib/sched_client.rb index f1c095d..fec4b14 100644 --- a/lib/sched_client.rb +++ b/lib/sched_client.rb @@ -3,12 +3,11 @@ # frozen_string_literal: true require 'rest-client' +require_relative 'constants' # sched client class class SchedClient - HOST = (ENV.key?('SCHED_HOST') ? ENV['SCHED_HOST'] : '172.17.0.1') - PORT = (ENV.key?('SCHED_PORT') ? ENV['SCHED_PORT'] : 3000).to_i - def initialize(host = HOST, port = PORT) + def initialize(host = SCHED_HOST, port = SCHED_PORT) @host = host @port = port end -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Jan '21

25 Jan '21

add limit for borrowing machine: runtime <= 10 days only jobs than runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..54affa1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 2