April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] service/scheduler: set time when update job
by Wu Zhende 12 Apr '21

12 Apr '21

Can use time in kibana to limit the search scope Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/elasticsearch_client.cr | 3 +++ 2 files changed, 7 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8691082..b6bb730 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -315,6 +315,10 @@ class Job return package_dir end + def set_time + self["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def set_upload_dirs self["upload_dirs"] = "#{result_root}#{get_package_dir}" end diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 58bed08..0e83ff1 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -36,6 +36,9 @@ class Elasticsearch::Client # caller should judge response["_id"] != nil def set_job_content(job : Job) + # time indicates the update time of each job event + job.set_time + response = get_job_content(job.id) if response["id"]? response = update(job.dump_to_json_any, job.id) -- 2.23.0

1 0

[PATCH v1 compass-ci] extract-stats: add Fiber.yield after spawn operate
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] bin/es-jobs-mapping.sh: add time to mapping
by Wu Zhende 12 Apr '21

12 Apr '21

Add the time of the date type to the ES mapping. "time" indicates the last update time of the job. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 502e099..9926fd8 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -88,6 +88,9 @@ else "errid": { "type": "text" }, + "time": { + "type": "date" + }, "category": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix error: Missing hash key: "os_mount"
by Cao Xueliang 12 Apr '21

12 Apr '21

When you use os_mount in set os_dir, you should set os_mount at first. Reference: 23db97c43d4dd06d8c7edec50523a66b7b781a46 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index d3c5cf9..8691082 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -157,11 +157,11 @@ class Job append_init_field() set_docker_os() set_user_lkp_src() + set_os_mount() set_os_dir() set_submit_date() set_pp_params() set_rootfs() - set_os_mount() set_result_root() set_result_service() set_depends_initrd() -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add check_public_network.sh
by Liu Yinsi 12 Apr '21

12 Apr '21

Check public network ok or not during locally deploy Compass CI, because users need to sparrow in a pure intranet environment. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/check_public_network.sh | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 lib/check_public_network.sh diff --git a/lib/check_public_network.sh b/lib/check_public_network.sh new file mode 100644 index 0000000..2d60b31 --- /dev/null +++ b/lib/check_public_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +public_network_ok() +{ + ping -c 3 -w 1 114.114.114.114 >/dev/null 2>&1 +} -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: add Fiber.yield after spawn operation
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH lkp-tests] sshd: always exec sshd before start test
by Wei Jihui 12 Apr '21

12 Apr '21

submit -i ssh.yaml job.yaml before: run_test $LKP_SRC/tests/wrapper job start_daemon $LKP_SRC/daemon/sshd run_test $LKP_SRC/tests/wrapper sl now: run_setup $LKP_SRC/setup/sshd run_test $LKP_SRC/tests/wrapper job run_test $LKP_SRC/tests/wrapper sleep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- {daemon => setup}/sshd | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {daemon => setup}/sshd (100%) diff --git a/daemon/sshd b/setup/sshd similarity index 100% rename from daemon/sshd rename to setup/sshd -- 2.23.0

1 0

[PATCH compass-ci] sched: we can search the job in es before return job id in submit job
by Cao Xueliang 09 Apr '21

09 Apr '21

use refresh=wait_for to make sure the job can be searched before consume the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/elasticsearch_client.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 3 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index f32256b..58bed08 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -142,6 +142,7 @@ class Elasticsearch::Client return @client.create( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => job_content, } @@ -152,6 +153,7 @@ class Elasticsearch::Client return @client.update( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => {:doc => job_content}, } diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 4f49394..41fa89f 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -76,6 +76,7 @@ class Sched if job_id begin job = @es.get_job(job_id.to_s) + @log.warn("job_is_nil, job id=#{job_id.to_s}") unless job rescue ex @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") @log.warn(ex.inspect_with_backtrace) -- 2.23.0

1 0

[PATCH v1 compass-ci 3/3] scheduler: add my start script for running myself scheduler
by Cao Xueliang 08 Apr '21

08 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/scheduler/my-start diff --git a/container/scheduler/my-start b/container/scheduler/my-start new file mode 100755 index 0000000..8d800f9 --- /dev/null +++ b/container/scheduler/my-start @@ -0,0 +1,85 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + REDIS_HOST + REDIS_PORT + ES_HOST + ES_PORT + OS_HTTP_HOST + OS_HTTP_PORT + INITRD_HTTP_HOST + INITRD_HTTP_PORT + TASKQUEUE_PORT + TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT + lab +] + +defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +env = docker_env(defaults) + +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] + +docker_rm "s001-alpine-#{SCHED_PORT}" + +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + +cmd = %W[ + docker run + --restart=always + --name s001-alpine-#{SCHED_PORT} + -d + -u 1090:1090 +] + bind_list + env + %W[ + -e CCI_REPOS=#{CCI_REPOS} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} + -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + -v /srv/initrd:/srv/initrd:ro + -v /srv/os:/srv/os:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + --log-opt tag=scheduler-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{SCHED_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/3] container: add scheduler-nginx for multiple scheduler nodes
by Cao Xueliang 08 Apr '21

08 Apr '21

use nginx as load balancing for multiple scheduler nodes. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler-nginx/Dockerfile | 8 +++++++ container/scheduler-nginx/build | 9 ++++++++ container/scheduler-nginx/nginx.conf | 29 +++++++++++++++++++++++++ container/scheduler-nginx/start | 32 ++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+) create mode 100644 container/scheduler-nginx/Dockerfile create mode 100755 container/scheduler-nginx/build create mode 100644 container/scheduler-nginx/nginx.conf create mode 100755 container/scheduler-nginx/start diff --git a/container/scheduler-nginx/Dockerfile b/container/scheduler-nginx/Dockerfile new file mode 100644 index 0000000..2cc430b --- /dev/null +++ b/container/scheduler-nginx/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM nginx:alpine + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +COPY nginx.conf /etc/nginx/ diff --git a/container/scheduler-nginx/build b/container/scheduler-nginx/build new file mode 100755 index 0000000..4b491d9 --- /dev/null +++ b/container/scheduler-nginx/build @@ -0,0 +1,9 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +docker_skip_rebuild "scheduler-nginx" + +docker build -t scheduler-nginx . diff --git a/container/scheduler-nginx/nginx.conf b/container/scheduler-nginx/nginx.conf new file mode 100644 index 0000000..c8c3783 --- /dev/null +++ b/container/scheduler-nginx/nginx.conf @@ -0,0 +1,29 @@ +# This is a default site configuration which will simply return 404, preventing +# chance access to any other virtualhost. +events { + worker_connections 65535; +} + +http{ + upstream sched { + server 172.17.0.1:3001; + server 172.17.0.1:3002; + server 172.17.0.1:3003; + } + + + server { + listen 3000 default_server; + listen [::]:3000 default_server; + + location / { + proxy_pass http://sched; + proxy_read_timeout 1814400; + } + + location = /404.html { + internal; + } + } +} + diff --git a/container/scheduler-nginx/start b/container/scheduler-nginx/start new file mode 100755 index 0000000..dc2dfbc --- /dev/null +++ b/container/scheduler-nginx/start @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_PORT +] + +defaults = relevant_defaults(names) +SCHED_PORT = defaults['SCHED_PORT'] +env = docker_env(defaults) + +docker_rm 'scheduler-nginx' + +cmd = %w[ + docker run + --name scheduler-nginx + --restart=always + -d +] + env + %W[ + -p #{SCHED_PORT}:#{SCHED_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + scheduler-nginx +] + +system(*cmd) -- 2.23.0

1 0