- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'new_ref_times'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 38 +++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..74eda85 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "source": [["x_params", $date1, ..., $date_today], +# ["new_ref_times", $times_1, ..., $times_today]] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..291fd9f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,41 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'source' => [['x_params'], ['new_ref_times']] + } + dev = 1 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['source'][0][dev] = day_s + echart['source'][1][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add a new os_mount named local
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 +- src/scheduler/kernel_params.cr | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..c013aa7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -344,7 +344,7 @@ class Job end # defaults to the 1st value - VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container"] + VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container", "local"] private def set_os_mount if is_docker_job? diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index dea9215..ed64bea 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,6 +18,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", + "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", } -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'y_params'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 39 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..6f85f35 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "x_params": [$date1, ..., $date_today], +# "y_params": [$times_1, ..., $times_today] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..1c29c88 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,42 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'x_params' => [], + 'y_params' => [] + } + dev = 0 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['x_params'][dev] = day_s + echart['y_params'][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] service/lifecycle: to process the machine that times out
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 51 ++++++++++++++++++++++++++++++++++++++ src/lifecycle/lifecycle.cr | 8 ++++++ 2 files changed, 59 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index af5cd07..75a18ec 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -4,6 +4,7 @@ require "kemal" require "yaml" +require "./scheduler_api" require "./web_env" require "../scheduler/elasticsearch_client" @@ -14,6 +15,7 @@ class Lifecycle @es = Elasticsearch::Client.new @env = env @log = env.log.as(JSONLogger) + @scheduler_api = SchedulerAPI.new end def alive(version) @@ -36,4 +38,53 @@ class Lifecycle } @es.search("testbox", query) end + + def get_timeout_testbox(size) + # why use "now+8h" + # The time zone in the ES database is 0 + # add 8 hours to the current time and convert it to time zone 0 + query = { + "size" => size, + "query" =>{ + "range" => { + "deadline" => { + "lt" => "now+8h" + } + } + } + } + + results = @es.search("testbox", query) + results["hits"]["hits"] + end + + def process_timeout_testbox + size = @env.params.query["size"]? || 20 + results = get_timeout_testbox(size) + return results unless results.is_a?(JSON::Any) + + results.as_a.each do |result| + hostname = result["_id"].to_s + hash = result["_source"] + close_lifecycle(hostname, hash, "timeout") + end + end + + # 1.reboot testbox, do in the fature + # 2.update testbox info in es + # 3.close job + def close_lifecycle(hostname, hash, close_reason) + state = hash["state"]? == "rebooting" ? hash["state"] : "forcibly_rebooting" + testbox_update_info = { + "deadline" => nil, + "state" => state + } + @es.update_tbox(hostname, testbox_update_info) + + # If the value of state is rebooting, + # the job has been closed and don't need to close it again + return if state == "rebooting" + + @scheduler_api.close_job(hash["job_id"], close_reason).status_message + end end diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr index 4154a27..13d36b7 100644 --- a/src/lifecycle/lifecycle.cr +++ b/src/lifecycle/lifecycle.cr @@ -29,4 +29,12 @@ module Cycle get "/get_running_testbox" do |env| env.lifecycle.get_running_testbox.to_json end + + # curl http://localhost:11311/process_timeout_testbox?size=10 + # size indicates the number of records to be processed at a time + get "/process_timeout_testbox" do |env| + env.lifecycle.process_timeout_testbox + + "done" + end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/scheduler_api.cr: added parameter "job_state"
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr index b8dd514..0fdcd12 100644 --- a/src/lib/scheduler_api.cr +++ b/src/lib/scheduler_api.cr @@ -10,10 +10,14 @@ class SchedulerAPI @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" end - def close_job(job_id) + def close_job(job_id, job_state = nil) + url = "/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}" + url += "&job_state=#{job_state}" if job_state + client = HTTP::Client.new(@host, port: @port) - response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + response = client.get(url) client.close() + return response end end -- 2.23.0

1 0

[PATCH compass-ci] lib/params_group.rb: change group_key value parse format
by Wang Yong 01 Feb '21

01 Feb '21

[before] os=openeuler/os_arch=aarch64/os_version=20.03/pp.sysbench-cpu.cpu_max_prime=20000/pp.sysbench-cpu.nr_threads=96/pp.sysbench-cpu.runtime=300/tbox_group=taishan200-2280-2s48p-256g [after] os=openeuler os_arch=aarch64 os_version=20.03 pp.sysbench-cpu.cpu_max_prime=20000 pp.sysbench-cpu.nr_threads=96 pp.sysbench-cpu.runtime=300 tbox_group=taishan200-2280-2s48p-256g it's easy to paste value and pass these to es-find Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/params_group.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 9be2cbe..2082c5d 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -124,7 +124,7 @@ end def get_group_key(group_params) group_str = group_params.each.map do |k, v| "#{k}=#{v}" - end.sort!.join('/') + end.sort!.join(' ') return group_str end -- 2.23.0

2 1

[PATCH v3 compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..6772961 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,26 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit most of the linux OSes. +# Avoid much repetitive testing work to adapt various OSes. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your local host. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] container/submit: submit file
by Luan Shengde 01 Feb '21

01 Feb '21

executable file for container: submit it can be run as the real command submit in the container. when you run it, it will calls the command 'submit' in the container. add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..20fdf61 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/submit: build file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 01 Feb '21

01 Feb '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..f957af8 --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +# it may take a long time if use default source address if in zh. +# you can delete the following step if you do not want to use it. +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +# it may take a long time, even break up if you use default source address for the gem. +# delete the following step if you do not want to use the default one. +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0