- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] extract-stats: add Fiber.yield after spawn operation
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH lkp-tests] sshd: always exec sshd before start test
by Wei Jihui 12 Apr '21

12 Apr '21

submit -i ssh.yaml job.yaml before: run_test $LKP_SRC/tests/wrapper job start_daemon $LKP_SRC/daemon/sshd run_test $LKP_SRC/tests/wrapper sl now: run_setup $LKP_SRC/setup/sshd run_test $LKP_SRC/tests/wrapper job run_test $LKP_SRC/tests/wrapper sleep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- {daemon => setup}/sshd | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {daemon => setup}/sshd (100%) diff --git a/daemon/sshd b/setup/sshd similarity index 100% rename from daemon/sshd rename to setup/sshd -- 2.23.0

1 0

[PATCH compass-ci] sched: we can search the job in es before return job id in submit job
by Cao Xueliang 09 Apr '21

09 Apr '21

use refresh=wait_for to make sure the job can be searched before consume the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/elasticsearch_client.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 3 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index f32256b..58bed08 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -142,6 +142,7 @@ class Elasticsearch::Client return @client.create( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => job_content, } @@ -152,6 +153,7 @@ class Elasticsearch::Client return @client.update( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => {:doc => job_content}, } diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 4f49394..41fa89f 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -76,6 +76,7 @@ class Sched if job_id begin job = @es.get_job(job_id.to_s) + @log.warn("job_is_nil, job id=#{job_id.to_s}") unless job rescue ex @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") @log.warn(ex.inspect_with_backtrace) -- 2.23.0

1 0

[PATCH v1 compass-ci 3/3] scheduler: add my start script for running myself scheduler
by Cao Xueliang 08 Apr '21

08 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/scheduler/my-start diff --git a/container/scheduler/my-start b/container/scheduler/my-start new file mode 100755 index 0000000..8d800f9 --- /dev/null +++ b/container/scheduler/my-start @@ -0,0 +1,85 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + REDIS_HOST + REDIS_PORT + ES_HOST + ES_PORT + OS_HTTP_HOST + OS_HTTP_PORT + INITRD_HTTP_HOST + INITRD_HTTP_PORT + TASKQUEUE_PORT + TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT + lab +] + +defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +env = docker_env(defaults) + +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] + +docker_rm "s001-alpine-#{SCHED_PORT}" + +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + +cmd = %W[ + docker run + --restart=always + --name s001-alpine-#{SCHED_PORT} + -d + -u 1090:1090 +] + bind_list + env + %W[ + -e CCI_REPOS=#{CCI_REPOS} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} + -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + -v /srv/initrd:/srv/initrd:ro + -v /srv/os:/srv/os:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + --log-opt tag=scheduler-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{SCHED_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/3] container: add scheduler-nginx for multiple scheduler nodes
by Cao Xueliang 08 Apr '21

08 Apr '21

use nginx as load balancing for multiple scheduler nodes. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler-nginx/Dockerfile | 8 +++++++ container/scheduler-nginx/build | 9 ++++++++ container/scheduler-nginx/nginx.conf | 29 +++++++++++++++++++++++++ container/scheduler-nginx/start | 32 ++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+) create mode 100644 container/scheduler-nginx/Dockerfile create mode 100755 container/scheduler-nginx/build create mode 100644 container/scheduler-nginx/nginx.conf create mode 100755 container/scheduler-nginx/start diff --git a/container/scheduler-nginx/Dockerfile b/container/scheduler-nginx/Dockerfile new file mode 100644 index 0000000..2cc430b --- /dev/null +++ b/container/scheduler-nginx/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM nginx:alpine + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +COPY nginx.conf /etc/nginx/ diff --git a/container/scheduler-nginx/build b/container/scheduler-nginx/build new file mode 100755 index 0000000..4b491d9 --- /dev/null +++ b/container/scheduler-nginx/build @@ -0,0 +1,9 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +docker_skip_rebuild "scheduler-nginx" + +docker build -t scheduler-nginx . diff --git a/container/scheduler-nginx/nginx.conf b/container/scheduler-nginx/nginx.conf new file mode 100644 index 0000000..c8c3783 --- /dev/null +++ b/container/scheduler-nginx/nginx.conf @@ -0,0 +1,29 @@ +# This is a default site configuration which will simply return 404, preventing +# chance access to any other virtualhost. +events { + worker_connections 65535; +} + +http{ + upstream sched { + server 172.17.0.1:3001; + server 172.17.0.1:3002; + server 172.17.0.1:3003; + } + + + server { + listen 3000 default_server; + listen [::]:3000 default_server; + + location / { + proxy_pass http://sched; + proxy_read_timeout 1814400; + } + + location = /404.html { + internal; + } + } +} + diff --git a/container/scheduler-nginx/start b/container/scheduler-nginx/start new file mode 100755 index 0000000..dc2dfbc --- /dev/null +++ b/container/scheduler-nginx/start @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_PORT +] + +defaults = relevant_defaults(names) +SCHED_PORT = defaults['SCHED_PORT'] +env = docker_env(defaults) + +docker_rm 'scheduler-nginx' + +cmd = %w[ + docker run + --name scheduler-nginx + --restart=always + -d +] + env + %W[ + -p #{SCHED_PORT}:#{SCHED_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + scheduler-nginx +] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 1/3] container: deploy three scheduler as the nginx upstream
by Cao Xueliang 08 Apr '21

08 Apr '21

use port 3001, 3002, 3003 as the scheduler port. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 41 +++++++++++++++++++++++---------------- sparrow/3-code/dev-env | 7 +++++++ 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index 8d800f9..b13b396 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -7,8 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - SCHED_HOST SCHED_PORT + SCHED_NODES REDIS_HOST REDIS_PORT ES_HOST @@ -35,36 +35,42 @@ defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' -defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +sched_nodes = defaults.delete('SCHED_NODES') +SCHED_PORT = defaults.delete('SCHED_PORT') + +sched_nodes.each do |node| + docker_rm "s001-alpine-#{node['port']}" +end env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' -SCHED_PORT = defaults['SCHED_PORT'] -SCHED_HOST = defaults['SCHED_HOST'] -docker_rm "s001-alpine-#{SCHED_PORT}" +sched_nodes.each do |node| + node_host = node['host'] + node_port = node['port'] -bind_list = [] -[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| - bind_list << '-p' - bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" -end + bind_list = [] + [node_host, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{node_port}:#{node_port}" + end -cmd = %W[ + cmd = %W[ docker run - --restart=always - --name s001-alpine-#{SCHED_PORT} + --name s001-alpine-#{node_port} -d -u 1090:1090 -] + bind_list + env + %W[ + ] + bind_list + env + %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e SCHED_PORT=#{node_port} + -v /srv/tmp/public:/c/cci/scheduler/public -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} @@ -78,8 +84,9 @@ cmd = %W[ --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_PORT} sch-ruby-a:v0.00d-#{SCHED_PORT} -] + ] -cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + cmd += ['sh', '-c', 'umask 002 && ./scheduler'] -system(*cmd) + system(*cmd) +end diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 7327083..fd529e7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -77,6 +77,13 @@ ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port +SCHED_NODES: + - host: $sched_host + port: 3001 + - host: $sched_host + port: 3002 + - host: $sched_host + port: 3003 LKP_SERVER: $server_ip GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST -- 2.23.0

1 0

[PATCH compass-ci] scheduler: add my start script for running myself scheduler
by Cao Xueliang 08 Apr '21

08 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/scheduler/my-start diff --git a/container/scheduler/my-start b/container/scheduler/my-start new file mode 100755 index 0000000..8d800f9 --- /dev/null +++ b/container/scheduler/my-start @@ -0,0 +1,85 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + REDIS_HOST + REDIS_PORT + ES_HOST + ES_PORT + OS_HTTP_HOST + OS_HTTP_PORT + INITRD_HTTP_HOST + INITRD_HTTP_PORT + TASKQUEUE_PORT + TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT + lab +] + +defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +env = docker_env(defaults) + +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] + +docker_rm "s001-alpine-#{SCHED_PORT}" + +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + +cmd = %W[ + docker run + --restart=always + --name s001-alpine-#{SCHED_PORT} + -d + -u 1090:1090 +] + bind_list + env + %W[ + -e CCI_REPOS=#{CCI_REPOS} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} + -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + -v /srv/initrd:/srv/initrd:ro + -v /srv/os:/srv/os:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + --log-opt tag=scheduler-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{SCHED_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 3/3] container: add scheduler-nginx for multiple scheduler nodes
by Cao Xueliang 08 Apr '21

08 Apr '21

use nginx as load balancing for multiple scheduler nodes. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler-nginx/Dockerfile | 8 +++++++ container/scheduler-nginx/build | 9 ++++++++ container/scheduler-nginx/nginx.conf | 29 +++++++++++++++++++++++++ container/scheduler-nginx/start | 32 ++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+) create mode 100644 container/scheduler-nginx/Dockerfile create mode 100755 container/scheduler-nginx/build create mode 100644 container/scheduler-nginx/nginx.conf create mode 100755 container/scheduler-nginx/start diff --git a/container/scheduler-nginx/Dockerfile b/container/scheduler-nginx/Dockerfile new file mode 100644 index 0000000..2cc430b --- /dev/null +++ b/container/scheduler-nginx/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM nginx:alpine + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +COPY nginx.conf /etc/nginx/ diff --git a/container/scheduler-nginx/build b/container/scheduler-nginx/build new file mode 100755 index 0000000..4b491d9 --- /dev/null +++ b/container/scheduler-nginx/build @@ -0,0 +1,9 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +docker_skip_rebuild "scheduler-nginx" + +docker build -t scheduler-nginx . diff --git a/container/scheduler-nginx/nginx.conf b/container/scheduler-nginx/nginx.conf new file mode 100644 index 0000000..c8c3783 --- /dev/null +++ b/container/scheduler-nginx/nginx.conf @@ -0,0 +1,29 @@ +# This is a default site configuration which will simply return 404, preventing +# chance access to any other virtualhost. +events { + worker_connections 65535; +} + +http{ + upstream sched { + server 172.17.0.1:3001; + server 172.17.0.1:3002; + server 172.17.0.1:3003; + } + + + server { + listen 3000 default_server; + listen [::]:3000 default_server; + + location / { + proxy_pass http://sched; + proxy_read_timeout 1814400; + } + + location = /404.html { + internal; + } + } +} + diff --git a/container/scheduler-nginx/start b/container/scheduler-nginx/start new file mode 100755 index 0000000..dc2dfbc --- /dev/null +++ b/container/scheduler-nginx/start @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_PORT +] + +defaults = relevant_defaults(names) +SCHED_PORT = defaults['SCHED_PORT'] +env = docker_env(defaults) + +docker_rm 'scheduler-nginx' + +cmd = %w[ + docker run + --name scheduler-nginx + --restart=always + -d +] + env + %W[ + -p #{SCHED_PORT}:#{SCHED_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + scheduler-nginx +] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 2/3] container: deploy three scheduler as the nginx upstream
by Cao Xueliang 08 Apr '21

08 Apr '21

use port 3001, 3002, 3003 as the scheduler port. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 43 +++++++++++++++++++++++---------------- sparrow/3-code/dev-env | 7 +++++++ 2 files changed, 32 insertions(+), 18 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index 8d800f9..70e6dbe 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -7,8 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - SCHED_HOST SCHED_PORT + SCHED_NODES REDIS_HOST REDIS_PORT ES_HOST @@ -35,36 +35,42 @@ defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' -defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +sched_nodes = defaults.delete('SCHED_NODES') +SCHED_PORT = defaults.delete('SCHED_PORT') + +sched_nodes.each do |node| + docker_rm "s001-alpine-#{node['port']}" +end env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' -SCHED_PORT = defaults['SCHED_PORT'] -SCHED_HOST = defaults['SCHED_HOST'] -docker_rm "s001-alpine-#{SCHED_PORT}" +sched_nodes.each do |node| + node_host = node['host'] + node_port = node['port'] -bind_list = [] -[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| - bind_list << '-p' - bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" -end + bind_list = [] + [node_host, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{node_port}:#{node_port}" + end -cmd = %W[ + cmd = %W[ docker run - --restart=always - --name s001-alpine-#{SCHED_PORT} + --name s001-alpine-#{node_port} -d -u 1090:1090 -] + bind_list + env + %W[ + ] + bind_list + env + %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e SCHED_PORT=#{node_port} + -v /srv/tmp/public:/c/cci/scheduler/public -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} @@ -76,10 +82,11 @@ cmd = %W[ -v /srv/os:/srv/os:ro --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=scheduler-#{SCHED_PORT} + --log-opt tag=scheduler-#{node_port} sch-ruby-a:v0.00d-#{SCHED_PORT} -] + ] -cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + cmd += ['sh', '-c', 'umask 002 && ./scheduler'] -system(*cmd) + system(*cmd) +end diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 7327083..fd529e7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -77,6 +77,13 @@ ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port +SCHED_NODES: + - host: $sched_host + port: 3001 + - host: $sched_host + port: 3002 + - host: $sched_host + port: 3003 LKP_SERVER: $server_ip GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST -- 2.23.0

1 0

[PATCH compass-ci 1/3] sparrow: add /srv/tmp/public in permission
by Cao Xueliang 08 Apr '21

08 Apr '21

The "/srv/tmp" stores temporary files, scheduler mount public dir in the dir for now. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/1-storage/permission | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 3d9bdfa..db33a93 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -12,6 +12,7 @@ chmod_dir=( /srv/cci/Maildir /srv/rabbitmq /c/git-repos + /srv/tmp/public ) chmod 775 "${chmod_dir[@]}" @@ -25,6 +26,7 @@ chown_dir=( /srv/cci/serial/fluentd-pos /srv/cci/libvirt-xml /srv/rabbitmq + /srv/tmp/public ) chown lkp:lkp "${chown_dir[@]}" -- 2.23.0

1 0

[PATCH compass-ci 1/2] extract-stats/stats_worker.cr: add errid to the job
by Wu Zhende 08 Apr '21

08 Apr '21

Save errid in es as type "text". In this way, we can search for errid by keyword. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/extract-stats/stats_worker.cr | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 56dad9e..e371bf3 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "../lib/etcd_client" @@ -8,6 +9,9 @@ require "./regression_client" require "./constants.cr" class StatsWorker + @@metric_failure = File.read("#{ENV["LKP_SRC"]}/etc/failure").strip.split("\n").as(Array(String)) + @@__is_failure_cache = Hash(String, Bool).new + def initialize @es = Elasticsearch::Client.new @etcd = EtcdClient.new @@ -45,6 +49,21 @@ class StatsWorker system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" end + def is_failure(stats_field) + if @@__is_failure_cache.has_key?(stats_field) + @@__is_failure_cache[stats_field] + else + @@__is_failure_cache[stats_field] = __is_failure(stats_field) + end + end + + def __is_failure(stats_field) + return false if stats_field.index(".time.") + return false if stats_field.index(".timestamp.") + return true if @@metric_failure.any? { |pattern| stats_field =~ %r{^#{pattern}} } + false + end + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" return unless File.exists?(stats_path) @@ -53,8 +72,13 @@ class StatsWorker JSON.parse(file) end + errid = Array(String).new + stats.as_h.keys.each do |k| + errid << k.to_s if is_failure(k.to_s) + end + update_content = Hash(String, Array(String) | Hash(String, JSON::Any)).new - update_content.merge!({"stats" => stats.as_h}) + update_content.merge!({"stats" => stats.as_h, "errid" => errid}) error_ids = get_error_ids_by_json(result_root) update_content.merge!({"error_ids" => error_ids}) unless error_ids.empty? -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin/es-jobs-mapping.sh: add errid
by Wu Zhende 08 Apr '21

08 Apr '21

The errid of the text type is added. Can use errid's keyword as a query. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 1c4aa43..28067e2 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -85,6 +85,9 @@ else "suite": { "type": "keyword" }, + "errid": { + "type": "text" + }, "category": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add submit_date field
by Lu Weitao 08 Apr '21

08 Apr '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 1c4aa43..fbfd4be 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -150,6 +150,10 @@ else "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, + "submit_date": { + "type": "date", + "format": "yyyy-MM-dd" + }, "user": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: when os_mount=local, add suffix '-iso' to os_version automically
by Xu Xijian 08 Apr '21

08 Apr '21

The version change example like follows: 1. 20.03 => 20.03-iso 2. 20.03-iso => 20.03-iso, no need to add iso suffix, just keep Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index f62073d..d83bbe5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -240,6 +240,7 @@ class Job end private def set_os_dir + self["os_version"] = "#{os_version}".chmop("-iso") + "-iso" if "#{self.os_mount}" == "local" self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

1 0

[PATCH v2 compass-ci] update container expose port
by Cui Yili 08 Apr '21

08 Apr '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/minio/start | 2 +- container/openresty-proxy-cache/start | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/minio/start b/container/minio/start index 1798cfa..2cbc9c8 100755 --- a/container/minio/start +++ b/container/minio/start @@ -7,7 +7,7 @@ docker_rm minio cmd=( docker run -d - -p 11321:9000 + -p 20014:9000 --name minio -v /etc/localtime:/etc/localtime:ro -v /srv/minio/:/srv/minio/ diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start index a1f095d..59e2a28 100755 --- a/container/openresty-proxy-cache/start +++ b/container/openresty-proxy-cache/start @@ -11,8 +11,8 @@ cmd=( docker run -d --name openresty-proxy-cache --restart=always - -p 11333:80 - -p 11334:443 + -p 20015:80 + -p 20016:443 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/initrd/tar:/srv/initrd/tar:ro -v /etc/localtime:/etc/localtime:ro -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: take git fail count into calculation of priority
by Li Yuanchao 07 Apr '21

07 Apr '21

If a repo git clone or git fetch failed many times, it would be put to the end of the queue. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index ad8847a..bcdb077 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -323,6 +323,11 @@ class MirrorMain @es_client.index(index: 'repo', type: '_doc', id: git_repo, body: body) end + def update_fail_count(git_repo, possible_new_refs) + @fork_stat[git_repo][:clone_fail_cnt] += 1 if possible_new_refs == -2 + @fork_stat[git_repo][:fetch_fail_cnt] += 1 if possible_new_refs == -1 + end + def update_stat_fetch(git_repo) @fork_stat[git_repo][:queued] = false offset_fetch = @fork_stat[git_repo][:offset_fetch] @@ -338,7 +343,6 @@ class MirrorMain end def update_stat_new_refs(git_repo) - @fork_stat[git_repo][:priority] += 1 offset_new_refs = @fork_stat[git_repo][:offset_new_refs] offset_new_refs = 0 if offset_new_refs >= 10 update_new_refs_info(git_repo, offset_new_refs) @@ -346,6 +350,7 @@ class MirrorMain def update_fork_stat(git_repo, possible_new_refs) update_stat_fetch(git_repo) + update_fail_count(git_repo, possible_new_refs) git_fail_log(git_repo, possible_new_refs) if possible_new_refs.negative? update_stat_new_refs(git_repo) if possible_new_refs.positive? && last_commit_new?(git_repo) new_repo_log(git_repo) if possible_new_refs == 2 @@ -416,7 +421,8 @@ class MirrorMain def get_fork_stat(git_repo) fork_stat = { queued: false, - priority: 0, + fetch_fail_cnt: 0, + clone_fail_cnt: 0, fetch_time: [], offset_fetch: 0, new_refs_time: [], @@ -581,18 +587,20 @@ class MirrorMain mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] - return old_pri + Math.cbrt(STEP_SECONDS) unless File.directory?(mirror_dir) + step = (@fork_stat[git_repo][:clone_fail_cnt] + 1) * Math.cbrt(STEP_SECONDS) + return old_pri + step unless File.directory?(mirror_dir) - return cal_priority(mirror_dir, old_pri) + return cal_priority(mirror_dir, old_pri, git_repo) end - def cal_priority(mirror_dir, old_pri) + def cal_priority(mirror_dir, old_pri, git_repo) last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i - return old_pri + Math.cbrt(STEP_SECONDS) if last_commit_time.zero? + step = (@fork_stat[git_repo][:fetch_fail_cnt] + 1) * Math.cbrt(STEP_SECONDS) + return old_pri + step if last_commit_time.zero? t = Time.now.to_i interval = t - last_commit_time - return old_pri + Math.cbrt(STEP_SECONDS) if interval <= 0 + return old_pri + step if interval <= 0 return old_pri + Math.cbrt(interval) end -- 2.23.0

1 0

[PATCH v2 compass-ci] container/mail-robot: fix bugs
by Luan Shengde 07 Apr '21

07 Apr '21

fix no file errors: it may throw error if the specified file not exists: Errno::ENOENT: No such file or directory @ rb_sysopen Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 798546a..d8cb4da 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -56,7 +56,7 @@ class ApplyAccount @es_port = ES_PORT # email info file for account issuers. - @account_issuer = YAML.load_file ENV['ACCOUNT_ISSUER'] + @account_issuer = File.exists?(ENV['ACCOUNT_ISSUER']) ? YAML.load_file(ENV['ACCOUNT_ISSUER']) : {} @my_info = {} end -- 2.23.0

1 0

[PATCH compass-ci] container/mail-robot: fix bugs
by Luan Shengde 07 Apr '21

07 Apr '21

fix no file errors: it may throw error if the specified file not exists: Errno::ENOENT: No such file or directory @ rb_sysopen Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 798546a..6b9bdfe 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -56,7 +56,11 @@ class ApplyAccount @es_port = ES_PORT # email info file for account issuers. - @account_issuer = YAML.load_file ENV['ACCOUNT_ISSUER'] + @account_issuer = if File.exists? ENV['ACCOUNT_ISSUER'] + YAML.load_file ENV['ACCOUNT_ISSUER'] + else + {} + end @my_info = {} end -- 2.23.0

2 4

[PATCH compass-ci] lib/git_mirror.rb: bigger step to change priority
by Li Yuanchao 07 Apr '21

07 Apr '21

Make the step to change priority of repo bigger, so bad repos would be put behind faster. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index b543f02..60a4b76 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -481,7 +481,7 @@ end # main thread class MirrorMain - WEEK_SECONDS = 604800 + MONTH_SECONDS = 2592000 def handle_feedback_new_refs(git_repo, feedback_info) return reload_fork_info(git_repo) if upstream_repo?(git_repo) @@ -584,18 +584,18 @@ class MirrorMain mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] - return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + return old_pri + Math.cbrt(MONTH_SECONDS) unless File.directory?(mirror_dir) return cal_priority(mirror_dir, old_pri) end def cal_priority(mirror_dir, old_pri) last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i - return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + return old_pri + Math.cbrt(MONTH_SECONDS) if last_commit_time.zero? t = Time.now.to_i interval = t - last_commit_time - return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + return old_pri + Math.cbrt(MONTH_SECONDS) if interval <= 0 return old_pri + Math.cbrt(interval) end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] lib/git_mirror.rb: add logs for kibana
by Li Yuanchao 07 Apr '21

07 Apr '21

Add logs when git clone fail, git fetch fail and clone a new repo. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 47 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 10 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6340400..ad8847a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -49,11 +49,18 @@ class GitMirror url = get_url(Array(url)[0]) 10.times do stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) - return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + return 2 if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") url = "git://#{url.split('://')[1]}" if stderr_443?(stderr) end - return false + return -2 + end + + def fetch_443(mirror_dir, fetch_info) + return unless stderr_443?(fetch_info) + + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) end def git_fetch(mirror_dir) @@ -67,12 +74,10 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + fetch_443(mirror_dir, fetch_info) + return -1 if fetch_info.include?('fatal') - if stderr_443?(fetch_info) - url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) - end - return fetch_info.include? '->' + return fetch_info.include?('->') ? 1 : 0 end def url_changed?(url, mirror_dir) @@ -196,8 +201,14 @@ class MirrorMain git_repo = feedback_info[:git_repo] return if check_submodule(git_repo) + # values of possible_new_refs: + # 2: git clone a new repo + # 1: git fetch and get new refs + # 0: git fetch and no new refs + # -1: git fetch fail + # -2: git clone fail update_fork_stat(git_repo, feedback_info[:possible_new_refs]) - return unless feedback_info[:possible_new_refs] + return if feedback_info[:possible_new_refs] < 1 handle_feedback_new_refs(git_repo, feedback_info) end @@ -335,7 +346,9 @@ class MirrorMain def update_fork_stat(git_repo, possible_new_refs) update_stat_fetch(git_repo) - update_stat_new_refs(git_repo) if possible_new_refs && last_commit_new?(git_repo) + git_fail_log(git_repo, possible_new_refs) if possible_new_refs.negative? + update_stat_new_refs(git_repo) if possible_new_refs.positive? && last_commit_new?(git_repo) + new_repo_log(git_repo) if possible_new_refs == 2 es_repo_update(git_repo) end end @@ -533,10 +546,24 @@ class MirrorMain end end + def new_repo_log(git_repo) + @log.info({ + msg: 'new repo', + repo: git_repo + }.to_json) + end + + def git_fail_log(git_repo, possible_new_refs) + msg = possible_new_refs == -1 ? 'git fetch fail' : 'git clone fail' + @log.info({ + msg: msg, + repo: git_repo + }.to_json) + end + def new_refs_log(git_repo, nr_new_branch) @log.info({ msg: 'new refs', - time: Time.now, repo: git_repo, nr_new_branch: nr_new_branch }.to_json) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: bigger step to change priority
by Li Yuanchao 07 Apr '21

07 Apr '21

Make the step to change priority of repo bigger, so bad repos would be put behind faster. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 1f4e5b5..6340400 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -468,7 +468,7 @@ end # main thread class MirrorMain - WEEK_SECONDS = 604800 + STEP_SECONDS = 2592000 def handle_feedback_new_refs(git_repo, feedback_info) return reload_fork_info(git_repo) if upstream_repo?(git_repo) @@ -554,18 +554,18 @@ class MirrorMain mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] - return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + return old_pri + Math.cbrt(STEP_SECONDS) unless File.directory?(mirror_dir) return cal_priority(mirror_dir, old_pri) end def cal_priority(mirror_dir, old_pri) last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i - return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + return old_pri + Math.cbrt(STEP_SECONDS) if last_commit_time.zero? t = Time.now.to_i interval = t - last_commit_time - return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + return old_pri + Math.cbrt(STEP_SECONDS) if interval <= 0 return old_pri + Math.cbrt(interval) end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-*-mapping.sh: fix illegal_argument_exception
by Lu Weitao 07 Apr '21

07 Apr '21

[Why] { "error":{ "root_cause":[{ "type":"illegal_argument_exception", "reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true." }], "type":"illegal_argument_exception", "reason":"The mapping definition cannot be nested under a type [_doc] unless include_type_name is set to true." }, "status":400 } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-accounts-mapping.sh | 34 ++++++++++----------- sbin/es-jobs-mapping.sh | 2 -- sbin/es-regression-mapping.sh | 16 +++++----- sbin/es-repo-mapping.sh | 56 +++++++++++++++++------------------ 4 files changed, 50 insertions(+), 58 deletions(-) diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index d04b0f3..c4ac2fc 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -12,24 +12,22 @@ else echo "begin create index." curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ "mappings": { - "_doc": { - "dynamic": false, - "properties": { - "my_commit_url": { - "type": "keyword" - }, - "my_token": { - "type": "keyword" - }, - "my_email": { - "type": "keyword" - }, - "my_name": { - "type": "keyword" - }, - "my_login_name": { - "type": "keyword" - } + "dynamic": false, + "properties": { + "my_commit_url": { + "type": "keyword" + }, + "my_token": { + "type": "keyword" + }, + "my_email": { + "type": "keyword" + }, + "my_name": { + "type": "keyword" + }, + "my_login_name": { + "type": "keyword" } } } diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index edc6159..1c4aa43 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -37,7 +37,6 @@ else echo "jobs index not exists, begin create index." curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { - "_doc": { "dynamic": false, "dynamic_templates": [ { @@ -166,7 +165,6 @@ else "format": "yyyy-MM-dd HH:mm:ss" } } - } } }' if [ $? -ne 0 ] diff --git a/sbin/es-regression-mapping.sh b/sbin/es-regression-mapping.sh index 8e04bdb..5ae0f71 100755 --- a/sbin/es-regression-mapping.sh +++ b/sbin/es-regression-mapping.sh @@ -10,15 +10,13 @@ else echo "begin create index." curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ "mappings": { - "_doc": { - "dynamic": false, - "properties": { - "error_id": { - "type": "keyword" - }, - "job_id": { - "type": "keyword" - } + "dynamic": false, + "properties": { + "error_id": { + "type": "keyword" + }, + "job_id": { + "type": "keyword" } } } diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh index 17f0986..290efb3 100755 --- a/sbin/es-repo-mapping.sh +++ b/sbin/es-repo-mapping.sh @@ -11,35 +11,33 @@ echo 'Start to create "repo" index.' curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' { "mappings": { - "_doc": { - "properties": { - "git_repo": { - "type": "keyword" - }, - "pkgbuild_repo": { - "type": "keyword" - }, - "url": { - "type": "keyword" - }, - "fetch_time": { - "type": "keyword" - }, - "new_refs_time": { - "type": "keyword" - }, - "offset_fetch": { - "type": "long" - }, - "offset_new_refs": { - "type": "long" - }, - "priority": { - "type": "long" - }, - "queued": { - "type": "boolean" - } + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" } } } -- 2.23.0

1 0

[PATCH compass-ci] lib/json_logger: unified time format
by Wu Zhende 07 Apr '21

07 Apr '21

The ES parses the time in this format to the date type. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/json_logger.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 2c33c2f..7a05f79 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -21,7 +21,7 @@ class JSONLogger < Logger logger_hash = { 'level' => severity.to_s, 'level_num' => level_num, - 'time' => datetime + 'time' => Time.now.strftime("%Y-%m-%dT%H:%M:%S.%3N+0800") } logger_hash['progname'] = progname if progname -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: add logs for kibana
by Li Yuanchao 07 Apr '21

07 Apr '21

Add logs when git clone fail, git fetch fail and clone a new repo. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 48 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 932a10e..60a4b76 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -49,11 +49,18 @@ class GitMirror url = get_url(Array(url)[0]) 10.times do stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) - return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + return 2 if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") url = "git://#{url.split('://')[1]}" if stderr_443?(stderr) end - return false + return -2 + end + + def fetch_443(mirror_dir, fetch_info) + return unless stderr_443?(fetch_info) + + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) end def git_fetch(mirror_dir) @@ -67,12 +74,10 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + fetch_443(mirror_dir, fetch_info) + return -1 if fetch_info.include?('fatal') - if stderr_443?(fetch_info) - url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) - end - return fetch_info.include? '->' + return fetch_info.include?('->') ? 1 : 0 end def url_changed?(url, mirror_dir) @@ -196,8 +201,14 @@ class MirrorMain git_repo = feedback_info[:git_repo] return if check_submodule(git_repo) + # values of possible_new_refs: + # 2: git clone a new repo + # 1: git fetch and get new refs + # 0: git fetch and no new refs + # -1: git fetch fail + # -2: git clone fail update_fork_stat(git_repo, feedback_info[:possible_new_refs]) - return unless feedback_info[:possible_new_refs] + return if feedback_info[:possible_new_refs] < 1 handle_feedback_new_refs(git_repo, feedback_info) end @@ -335,7 +346,9 @@ class MirrorMain def update_fork_stat(git_repo, possible_new_refs) update_stat_fetch(git_repo) - update_stat_new_refs(git_repo) if possible_new_refs && last_commit_new?(git_repo) + git_fail_log(git_repo, possible_new_refs) if possible_new_refs.negative? + update_stat_new_refs(git_repo) if possible_new_refs.positive? && last_commit_new?(git_repo) + new_repo_log(git_repo) if possible_new_refs == 2 es_repo_update(git_repo) end end @@ -533,6 +546,23 @@ class MirrorMain end end + def new_repo_log(git_repo) + @log.info({ + msg: 'new repo', + time: Time.now, + repo: git_repo + }.to_json) + end + + def git_fail_log(git_repo, possible_new_refs) + msg = possible_new_refs == -1 ? 'git fetch fail' : 'git clone fail' + @log.info({ + msg: msg, + time: Time.now, + repo: git_repo + }.to_json) + end + def new_refs_log(git_repo, nr_new_branch) @log.info({ msg: 'new refs', -- 2.23.0

1 0

[PATCH compass-ci] lib/lifecycle: add some logs
by Wu Zhende 07 Apr '21

07 Apr '21

Help locate problems. Collects statistics on the number and information of close jobs. Collects statistics on the number and information of reboot machines. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 43 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 0e6cb82..e56a681 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -43,7 +43,10 @@ class Lifecycle def alive(version) "Lifecycle Alive! The time is #{Time.local}, version = #{version}" rescue e - @log.warn(e) + @log.warn({ + "resource" => "/alive", + "message" => e.to_s + }.to_json) end def init_from_es @@ -68,6 +71,11 @@ class Lifecycle deal_match_job(testbox, machine["job_id"].to_s) end + rescue e + @log.warn({ + "resource" => "init_from_es", + "message" => e.to_s + }.to_json) end def deal_match_job(testbox, job_id) @@ -129,6 +137,11 @@ class Lifecycle end @mq.ch.basic_ack(msg.delivery_tag) end + rescue e + @log.warn({ + "resource" => "mq_event_loop", + "message" => e.to_s + }.to_json) end def on_other_job(event) @@ -253,6 +266,11 @@ class Lifecycle next if dead_job_id && deadline <= Time.local sleep_until(deadline) + rescue e + @log.warn({ + "resource" => "timeout_job_loop", + "message" => e.to_s + }.to_json) end end @@ -264,6 +282,11 @@ class Lifecycle next if dead_machine_name && deadline <= Time.local sleep_until(deadline) + rescue e + @log.warn({ + "resource" => "timeout_machine_loop", + "message" => e.to_s + }.to_json) end end @@ -306,6 +329,12 @@ class Lifecycle def close_job(job_id, reason) @jobs.delete(job_id) spawn @scheduler_api.close_job(job_id, reason, "lifecycle") + @log.info({ + "job_id" => job_id, + "state" => "close", + "reason" => reason, + "type" => "job" + }.to_json) end def reboot_crash_machine(testbox, event) @@ -314,7 +343,7 @@ class Lifecycle return unless machine return unless event["time"].to_s.bigger_than?(machine["time"]?) - reboot_machine(testbox, machine) + reboot_machine(testbox, machine, "crash") end def reboot_timeout_machine(testbox) @@ -329,10 +358,10 @@ class Lifecycle deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) return if Time.local < deadline - reboot_machine(testbox, machine) + reboot_machine(testbox, machine, "timeout") end - def reboot_machine(testbox, machine) + def reboot_machine(testbox, machine, reason) mq_queue = get_machine_reboot_queue(testbox) machine.as_h.delete("history") machine.as_h["testbox"] = JSON::Any.new(testbox) @@ -341,6 +370,12 @@ class Lifecycle machine["state"] = "rebooting_queue" machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") @es.update_tbox(testbox, machine.as_h) + @log.info({ + "type" => "testbox", + "reason" => reason, + "testbox" => testbox, + "state" => "reboot" + }.to_json) end def get_machine_reboot_queue(testbox) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/rpm-repo: add a new docker service
by Li Ping 07 Apr '21

07 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/Dockerfile | 10 +++++ container/rpm-repo/build | 6 +++ container/rpm-repo/start | 41 ++++++++++++++++++++ lib/rpm-repo.rb | 73 +++++++++++++++++++++++++++++++++++ 4 files changed, 130 insertions(+) create mode 100644 container/rpm-repo/Dockerfile create mode 100755 container/rpm-repo/build create mode 100755 container/rpm-repo/start create mode 100755 lib/rpm-repo.rb diff --git a/container/rpm-repo/Dockerfile b/container/rpm-repo/Dockerfile new file mode 100644 index 0000000..588828a --- /dev/null +++ b/container/rpm-repo/Dockerfile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fedora:latest + +RUN yum install -y createrepo ruby + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install bunny diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . diff --git a/container/rpm-repo/start b/container/rpm-repo/start new file mode 100755 index 0000000..08fa434 --- /dev/null +++ b/container/rpm-repo/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +DOCKER_CCI = '/c/compass-ci' + +names = Set.new %w[ + MQ_HOST + MQ_PORT + LKP_SERVER +] + +defaults = relevant_defaults(names) +defaults['MQ_HOST'] ||= defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['MQ_PORT'] ||= 5672 +env = docker_env(defaults) + +docker_rm "update-repodata" + +cmd = %W[ + docker run + --restart=always + --name update-repodata + -u 1090:1090 +] + env + %W[ + -d + -e CCI_SRC=#{DOCKER_CCI} + -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} + -v /etc/localtime:/etc/localtime:ro + -v /srv/rpm:/srv/rpm + -w /c/compass-ci/lib + fedora:update-repodata +] + +cmd += ['sh', '-c', 'umask 002 && ./rpm-repo.rb'] +system(*cmd) diff --git a/lib/rpm-repo.rb b/lib/rpm-repo.rb new file mode 100755 index 0000000..658a527 --- /dev/null +++ b/lib/rpm-repo.rb @@ -0,0 +1,73 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'fileutils' +require 'json' +require './mq_client' + +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || '172.17.0.1' +MQ_PORT = ENV['MQ_PORT'] || 5672 + +# --------------------------------------------------------------------------------------------------------------------------------------- +# end_user can use cmd: +# curl -H 'Content-Type: Application/json' -XPOST 'localhost:10003/upload' \ +# -d '{"upload_rpms": ["/srv/rpm/upload/openeuler-20.03-LTS/compat-centos8/centos-appstream/aarch64/Packages/*.rpm"]}' +# +# the post data would be stored in mq queue "update_repo" through update-repo service +# --------------------------------------------------------------------------------------------------------------------------------------- +# +# HandleRepo used to handle the mq queue "update_repo" +# +# Example items in @mq.queue "update_repo": { "upload_rpms" => ["/srv/rpm/upload/**/Packages/*.rpm", "/srv/rpm/upload/**/source/*.rpm"]} +# handle_new_rpm +# move /srv/rpm/upload/**/*.rpm to /srv/rpm/testing/**/*.rpm +# update /srv/rpm/testing/**/repodate +# update_pub_dir +# copy /srv/rpm/testing/**/*.rpm to /srv/rpm/pub/**/*.rpm +# change /srv/rpm/pub/**/repodate +# @mq.ack(info) +class HandleRepo + def initialize + @mq = MQClient.new(MQ_HOST, MQ_PORT) + @update = [] + end + + def handle_new_rpm + queue = @mq.queue("update_repo") + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + rpm_info = JSON.parse(msg) + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + + dest = File.join(rpm_path.to_s, File.basename(rpm)) + @update << dest + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + update_pub_dir + @mq.ack(info) + end + end + + def update_pub_dir + @update.each do |rpm| + pub_path = File.dirname(rpm).sub("testing", "pub") + FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) + + dest = File.join(pub_path, File.basename(rpm)) + FileUtils.cp(rpm, dest) + + repodata_dest = File.join(File.dirname(pub_path), "repodata") + repodata_src = File.dirname(rpm).sub("Packages", "repodata") + + FileUtils.rm_r(repodata_dest) if Dir.exist?(repodata_dest) + FileUtils.cp_r(repodata_src, File.dirname(repodata_dest)) + end + end +end + +hr = HandleRepo.new +hr.handle_new_rpm -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] src/lib/updaterepo.cr: send messages to update-repo queue
by Li Ping 07 Apr '21

07 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/updaterepo.cr | 65 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 src/lib/updaterepo.cr diff --git a/src/lib/updaterepo.cr b/src/lib/updaterepo.cr new file mode 100644 index 0000000..78d6a52 --- /dev/null +++ b/src/lib/updaterepo.cr @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "json" +require "kemal" +require "./json_logger" +require "./mq" + +# ------------------------------------------------------------------------------------------------------------------------------------- +# end_user: +# - restful API [post "/upload"] to post data {"upload_rpms": ["/srv/rpm/upload/**/Packages/*.rpm", "/srv/rpm/upload/**/source/*.rpm"]} +# +# ------------------------------------------------------------------------------------------------------------------------------------- +# Repo: +# - get json formated request data +# - use @mq.queue("update_repo") publish data +# - json formated data stored in the @mq.queue("update_repo") +# +class Repo + def initialize(env : HTTP::Server::Context) + @mq = MQClient.instance + @env = env + @log = JSONLogger.new + end + + def upload_repo + begin + body = @env.request.body.not_nil!.gets_to_end + data = JSON.parse(body.to_s).as_h? + rescue e + @log.warn(e) + end + @log.info(data.to_json) + + begin + check_params_complete(data) + mq_publish(data) + rescue e + @log.warn(e) + response = { "errcode" => "101", "errmsg" => "upload rpm failed" } + @log.warn(response.to_json) + end + end + + def mq_publish(data) + mq_msg = data + spawn mq_publish_check("update_repo", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def check_params_complete(params) + params = params.not_nil! + tmp_hash = {"errcode" => "101", "errmsg" => "no upload_rpms params"}.to_json + raise tmp_hash unless params["upload_rpms"]? + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] add update-repo service
by Li Ping 07 Apr '21

07 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/web_env.cr | 9 +++++++++ src/updaterepo.cr | 17 +++++++++++++++++ src/updaterepo/constants.cr | 4 ++++ src/updaterepo/updaterepo.cr | 16 ++++++++++++++++ 4 files changed, 46 insertions(+) create mode 100644 src/updaterepo.cr create mode 100644 src/updaterepo/constants.cr create mode 100644 src/updaterepo/updaterepo.cr diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index b3a90a8..d47e7e5 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -4,6 +4,7 @@ require "./sched" require "./lifecycle" require "./json_logger" +require "./updaterepo" class HTTP::Server # Instances of this class are passed to an `HTTP::Server` handler. @@ -31,5 +32,13 @@ class HTTP::Server def create_lifecycle @lifecycle = Lifecycle.new(self) end + + def repo + @repo ||= create_repo + end + + def create_repo + @repo = Repo.new(self) + end end end diff --git a/src/updaterepo.cr b/src/updaterepo.cr new file mode 100644 index 0000000..05fa4fe --- /dev/null +++ b/src/updaterepo.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "./updaterepo/updaterepo" +require "./updaterepo/constants" +require "./lib/json_logger" + + +module Updaterepo + log = JSONLogger.new + + begin + Kemal.run(REPO_PORT) + rescue e + log.error(e) + end +end diff --git a/src/updaterepo/constants.cr b/src/updaterepo/constants.cr new file mode 100644 index 0000000..25d9978 --- /dev/null +++ b/src/updaterepo/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +REPO_PORT = (ENV.has_key?("REPO_PORT") ? ENV["REPO_PORT"] : 10003).to_i32 diff --git a/src/updaterepo/updaterepo.cr b/src/updaterepo/updaterepo.cr new file mode 100644 index 0000000..7ad9d20 --- /dev/null +++ b/src/updaterepo/updaterepo.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/updaterepo" +require "../lib/json_logger" + +module Updaterepo + VERSION = "0.1.0" + post "/upload" do |env| + env.repo.upload_repo + "done" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/updaterepo: add a new docker service
by Li Ping 07 Apr '21

07 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ container/updaterepo/build | 14 ++++++++++++++ container/updaterepo/start | 24 ++++++++++++++++++++++++ 3 files changed, 59 insertions(+) create mode 100644 container/updaterepo/Dockerfile create mode 100755 container/updaterepo/build create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..4ac1925 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 10003:10003 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci] sched: fix split jobs by email
by Cao Xueliang 07 Apr '21

07 Apr '21

The queue is "/queues/sched/$queue/ready/$subqueue/$id", $subqueue is the email, so we should get the 5 index when split by "/". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 99c577f..4f49394 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -117,7 +117,7 @@ class Sched def split_jobs_by_email(jobs) hash = Hash(String, Array(Etcd::Model::Kv)).new jobs.each do |job| - key = job.key.split("/")[4] + key = job.key.split("/")[5] if hash.has_key?(key) hash[key] << job else -- 2.23.0

1 0

[PATCH v1 compass-ci] sparrow: etcd-server no need to deploy if it's running
by Cao Xueliang 07 Apr '21

07 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/4-docker/buildall | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3611158..e228153 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -58,10 +58,15 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == "ssh-r" ] && { - container_id=$(docker ps -aqf name="ssh_r") - [ -n "$container_id" ] && exit - } + declare -A dic + local dic=(['ssh-r']='ssh_r' ['etcd']='etcd-server') + for item in ${!dic[@]} + do + [ "$container_name" == ${item} ] && { + container_id=$(docker ps -aqf name=${dic[${item}]}) + [ -n "$container_id" ] && exit + } + done [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci] sparrow: etcd-server no need to deploy if it's running
by Cao Xueliang 07 Apr '21

07 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/4-docker/buildall | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3611158..532f2b4 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -58,10 +58,15 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == "ssh-r" ] && { - container_id=$(docker ps -aqf name="ssh_r") - [ -n "$container_id" ] && exit - } + declare -A dic + local dic=(['ssh-r']='ssh_r' ['etcd']='etcd-server') + for item in ${!dic[@]} + do + [ "$container_name" == ${item} ] && { + container_id=$(docker ps -aqf name=${dic[${item}}) + [ -n "$container_id" ] && exit + } + done [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run -- 2.23.0

1 1

[PATCH lab-z9] add some host files
by Zhang Dewan 06 Apr '21

06 Apr '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a16 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a87 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a99 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a122 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a125 | 23 +++++++++++++++++++++++ 5 files changed, 105 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a16 create mode 100644 hosts/taishan200-2280-2s48p-256g--a87 create mode 100644 hosts/taishan200-2280-2s48p-256g--a99 create mode 100644 hosts/taishan200-2280-2s64p-256g--a122 create mode 100644 hosts/taishan200-2280-2s64p-256g--a125 diff --git a/hosts/taishan200-2280-2s48p-256g--a16 b/hosts/taishan200-2280-2s48p-256g--a16 new file mode 100644 index 0000000..7aba0af --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a16 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bf271 + - /dev/disk/by-id/scsi-350000399b88bf1a9 + - /dev/disk/by-id/scsi-350000399b88bf0d5 + - /dev/disk/by-id/scsi-350000399b88bf385 +mac_addr: + - 84:46:fe:3d:a2:17 + - 84:46:fe:3d:a2:18 + - 84:46:fe:3d:a2:19 + - 84:46:fe:3d:a2:1a + - 84:46:fe:3d:a2:1b + - 84:46:fe:3d:a2:1c + - 84:46:fe:3d:a2:1d + - 84:46:fe:3d:a2:1e +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a87 b/hosts/taishan200-2280-2s48p-256g--a87 new file mode 100644 index 0000000..db88e7e --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a87 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a6b55 + - /dev/disk/by-id/scsi-350000399886ad4a5 + - /dev/disk/by-id/scsi-350000399886a7139 + - /dev/disk/by-id/scsi-350000399886acb21 +mac_addr: + - 44:67:47:c9:eb:66 + - 44:67:47:c9:eb:67 + - 44:67:47:c9:eb:68 + - 44:67:47:c9:eb:69 + - 44:67:47:c9:eb:6a + - 44:67:47:c9:eb:6b + - 44:67:47:c9:eb:6c + - 44:67:47:c9:eb:6d +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.8.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a99 b/hosts/taishan200-2280-2s48p-256g--a99 new file mode 100644 index 0000000..205b5cd --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a99 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b5f81 + - /dev/disk/by-id/scsi-350000399886a7199 + - /dev/disk/by-id/scsi-350000399886a6ea9 + - /dev/disk/by-id/scsi-35000c500bd66d04b +mac_addr: + - 5c:e8:83:73:16:43 + - 5c:e8:83:73:16:44 + - 5c:e8:83:73:16:45 + - 5c:e8:83:73:16:46 + - 5c:e8:83:73:16:47 + - 5c:e8:83:73:16:48 + - 5c:e8:83:73:16:49 + - 5c:e8:83:73:16:4a +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.4 diff --git a/hosts/taishan200-2280-2s64p-256g--a122 b/hosts/taishan200-2280-2s64p-256g--a122 new file mode 100644 index 0000000..5e3d9b7 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a122 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-3500003998881eb0d + - /dev/disk/by-id/scsi-35000039988816c75 +mac_addr: + - 44:a1:91:9d:d8:25 + - 44:a1:91:9d:d8:26 + - 44:a1:91:9d:d8:27 + - 44:a1:91:9d:d8:28 + - 44:a1:91:e5:7b:01 + - 44:a1:91:e5:7b:02 + - 44:a1:91:e5:7b:03 + - 44:a1:91:e5:7b:04 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.3 diff --git a/hosts/taishan200-2280-2s64p-256g--a125 b/hosts/taishan200-2280-2s64p-256g--a125 new file mode 100644 index 0000000..5c5862e --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a125 @@ -0,0 +1,23 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988707ca1 + - scsi-35000039988707ca10 + - scsi-35000039988707ca1-part* + - /dev/disk/by-id/scsi-35000039988707d25-part1 + - /dev/disk/by-id/scsi-35000039988707d25-part2 + - /dev/disk/by-id/scsi-35000039988707d25-part3 +mac_addr: + - 44:67:47:85:f4:65 + - 44:67:47:85:f4:66 + - 44:67:47:85:f4:67 + - 44:67:47:85:f4:68 + - 64:2c:ac:df:06:50 + - 64:2c:ac:df:06:51 + - 64:2c:ac:df:06:52 + - 64:2c:ac:df:06:53 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.6 -- 2.23.0

2 1

[PATCH compass-ci 4/4] add update-repo service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/web_env.cr | 9 +++++++++ src/updaterepo.cr | 17 +++++++++++++++++ src/updaterepo/constants.cr | 4 ++++ src/updaterepo/updaterepo.cr | 16 ++++++++++++++++ 4 files changed, 46 insertions(+) create mode 100644 src/updaterepo.cr create mode 100644 src/updaterepo/constants.cr create mode 100644 src/updaterepo/updaterepo.cr diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index b3a90a8..d47e7e5 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -4,6 +4,7 @@ require "./sched" require "./lifecycle" require "./json_logger" +require "./updaterepo" class HTTP::Server # Instances of this class are passed to an `HTTP::Server` handler. @@ -31,5 +32,13 @@ class HTTP::Server def create_lifecycle @lifecycle = Lifecycle.new(self) end + + def repo + @repo ||= create_repo + end + + def create_repo + @repo = Repo.new(self) + end end end diff --git a/src/updaterepo.cr b/src/updaterepo.cr new file mode 100644 index 0000000..05fa4fe --- /dev/null +++ b/src/updaterepo.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "./updaterepo/updaterepo" +require "./updaterepo/constants" +require "./lib/json_logger" + + +module Updaterepo + log = JSONLogger.new + + begin + Kemal.run(REPO_PORT) + rescue e + log.error(e) + end +end diff --git a/src/updaterepo/constants.cr b/src/updaterepo/constants.cr new file mode 100644 index 0000000..25d9978 --- /dev/null +++ b/src/updaterepo/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +REPO_PORT = (ENV.has_key?("REPO_PORT") ? ENV["REPO_PORT"] : 10003).to_i32 diff --git a/src/updaterepo/updaterepo.cr b/src/updaterepo/updaterepo.cr new file mode 100644 index 0000000..7ad9d20 --- /dev/null +++ b/src/updaterepo/updaterepo.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/updaterepo" +require "../lib/json_logger" + +module Updaterepo + VERSION = "0.1.0" + post "/upload" do |env| + env.repo.upload_repo + "done" + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] src/lib/updaterepo.cr: send messages to update-repo queue
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/updaterepo.cr | 54 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 src/lib/updaterepo.cr diff --git a/src/lib/updaterepo.cr b/src/lib/updaterepo.cr new file mode 100644 index 0000000..3587de5 --- /dev/null +++ b/src/lib/updaterepo.cr @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "json" +require "kemal" +require "./json_logger" +require "./mq" + +class Repo + def initialize(env : HTTP::Server::Context) + @mq = MQClient.instance + @env = env + @log = JSONLogger.new + end + + def upload_repo + begin + body = @env.request.body.not_nil!.gets_to_end + data = JSON.parse(body.to_s).as_h? + rescue e + @log.error(e) + end + + begin + check_params_complete(data) + mq_publish(data) + rescue e + response = { "errcode" => "101", "errmsg" => "upload rpm failed" } + @log.info(response.to_json) + puts "error message: #{e.message}" + end + end + + def mq_publish(data) + mq_msg = data + spawn mq_publish_check("update_repo", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def check_params_complete(params) + params = params.not_nil! + tmp_hash = {"errcode" => "101", "errmsg" => "no upload_rpms params"}.to_json + raise tmp_hash unless params["upload_rpms"]? + end +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/updaterepo: add a new docker service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ container/updaterepo/build | 14 ++++++++++++++ container/updaterepo/start | 24 ++++++++++++++++++++++++ 3 files changed, 59 insertions(+) create mode 100644 container/updaterepo/Dockerfile create mode 100755 container/updaterepo/build create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..4ac1925 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 10003:10003 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/rpm-repo: add a new docker service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/Dockerfile | 10 ++++++ container/rpm-repo/build | 6 ++++ container/rpm-repo/start | 41 ++++++++++++++++++++++ lib/rpm-repo.rb | 65 +++++++++++++++++++++++++++++++++++ 4 files changed, 122 insertions(+) create mode 100644 container/rpm-repo/Dockerfile create mode 100755 container/rpm-repo/build create mode 100755 container/rpm-repo/start create mode 100755 lib/rpm-repo.rb diff --git a/container/rpm-repo/Dockerfile b/container/rpm-repo/Dockerfile new file mode 100644 index 0000000..588828a --- /dev/null +++ b/container/rpm-repo/Dockerfile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fedora:latest + +RUN yum install -y createrepo ruby + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install bunny diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . diff --git a/container/rpm-repo/start b/container/rpm-repo/start new file mode 100755 index 0000000..08fa434 --- /dev/null +++ b/container/rpm-repo/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +DOCKER_CCI = '/c/compass-ci' + +names = Set.new %w[ + MQ_HOST + MQ_PORT + LKP_SERVER +] + +defaults = relevant_defaults(names) +defaults['MQ_HOST'] ||= defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['MQ_PORT'] ||= 5672 +env = docker_env(defaults) + +docker_rm "update-repodata" + +cmd = %W[ + docker run + --restart=always + --name update-repodata + -u 1090:1090 +] + env + %W[ + -d + -e CCI_SRC=#{DOCKER_CCI} + -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} + -v /etc/localtime:/etc/localtime:ro + -v /srv/rpm:/srv/rpm + -w /c/compass-ci/lib + fedora:update-repodata +] + +cmd += ['sh', '-c', 'umask 002 && ./rpm-repo.rb'] +system(*cmd) diff --git a/lib/rpm-repo.rb b/lib/rpm-repo.rb new file mode 100755 index 0000000..f516d3f --- /dev/null +++ b/lib/rpm-repo.rb @@ -0,0 +1,65 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'fileutils' +require 'json' +require './mq_client' + +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || '172.17.0.1' +MQ_PORT = ENV['MQ_PORT'] || 5672 + +# used to handle the mq queue "update_repo" +# +# Example items in @mq.queue "update_repo": { "upload_rpms" => ["/srv/rpm/upload/**/Packages/*.rpm", "/srv/rpm/upload/**/source/*.rpm"]} +# handle_new_rpm +# move /srv/rpm/upload/**/*.rpm to /srv/rpm/testing/**/*.rpm +# update /srv/rpm/testing/**/repodate +# update_pub_dir +# copy /srv/rpm/testing/**/*.rpm to /srv/rpm/pub/**/*.rpm +# change /srv/rpm/pub/**/repodate +# @mq.ack(info) +class HandleRepo + def initialize + @mq = MQClient.new(MQ_HOST, MQ_PORT) + @update = [] + end + + def handle_new_rpm + queue = @mq.queue("update_repo") + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + rpm_info = JSON.parse(msg) + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + + dest = File.join(rpm_path.to_s, File.basename(rpm)) + @update << dest + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + update_pub_dir + @mq.ack(info) + end + end + + def update_pub_dir + @update.each do |rpm| + pub_path = File.dirname(rpm).sub("testing", "pub") + FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) + + dest = File.join(pub_path, File.basename(rpm)) + FileUtils.cp(rpm, dest) + + repodata_dest = File.join(File.dirname(pub_path), "repodata") + repodata_src = File.dirname(rpm).sub("Packages", "repodata") + + FileUtils.rm_r(repodata_dest) if Dir.exist?(repodata_dest) + FileUtils.cp_r(repodata_src, File.dirname(repodata_dest)) + end + end +end + +hr = HandleRepo.new +hr.handle_new_rpm -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add build-pkg.rb to generate error_id with message
by Lin Jiaxin 06 Apr '21

06 Apr '21

output like this: "build-pkg.include/common/ist.h:warning:unused-parameter‘count’[-Wunused-parameter]": 1, "build-pkg.include/common/ist.h:warning:unused-parameter‘count’[-Wunused-parameter].message": [ "include/common/ist.h:488:74: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str(char *dst, const struct ist src, size_t count)\n ^~~~~\n", "include/common/ist.h:533:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_lc(char *dst, const struct ist src, size_t count )\n ^~~~~\n", "include/common/ist.h:578:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_uc(char *dst, const struct ist src, size_t count )\n ^~~~~\n" ] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg.rb | 74 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 stats/build-pkg.rb diff --git a/stats/build-pkg.rb b/stats/build-pkg.rb new file mode 100755 index 00000000..4515bf23 --- /dev/null +++ b/stats/build-pkg.rb @@ -0,0 +1,74 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require 'set' +require "#{LKP_SRC}/lib/common" + +class ErrorMessages + def initialize + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages(log_lines) + log_lines.each do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end +end + +def build_pkg(log_lines) + message = {} + ErrorMessages.new.obtain_error_messages(log_lines).each do |k, v| + message[k] = '1' + message["#{k}.message"] = v.to_a + end + + return message +end -- 2.23.0

1 0

[PATCH compass-ci] sched: consume job by priority
by Cao Xueliang 06 Apr '21

06 Apr '21

before: consume job by shuffle after: our priority is reflected in email account for now, consume job by priority and the delimiter@localhost has the highest priority Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 40 ++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 01b6f1d..a0e5220 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -110,15 +110,47 @@ class Sched def consume_by_list(queues) jobs, revision = get_history_jobs(queues) - jobs.shuffle! - while true - return nil, revision if jobs.empty? + email_jobs = split_jobs_by_email(jobs) + loop do + return nil, revision if email_jobs.empty? + + job = pop_job_by_priority(email_jobs) + return nil, revision unless job - job = jobs.delete_at(0) return job, revision if ready2process(job) end end + def split_jobs_by_email(jobs) + hash = Hash(String, Array(Etcd::Model::Kv)).new + jobs.each do |job| + key = job.key.split("/")[4] + if hash.has_key?(key) + hash[key] << job + else + hash[key] = [job] + end + end + + return hash + end + + def pop_job_by_priority(email_jobs) + delimiter = "delimiter@localhost" + if email_jobs.has_key?(delimiter) + return email_jobs[delimiter].delete_at(0) unless email_jobs[delimiter].empty? + + email_jobs.delete(delimiter) + end + + keys = rand_queues(email_jobs.keys) + keys.each do |key| + return email_jobs[key].delete_at(0) unless email_jobs[key].empty? + + email_jobs.delete(key) + end + end + def get_history_jobs(queues) revisions = [] of Int64 ec = EtcdClient.new -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: fix redirect issue
by Lu Weitao 06 Apr '21

06 Apr '21

[Why] the old srv-http-result container port is: 20007:11300 when access srv-http-result like: http://$host:20007/result # (without '/' at the end) the nginx will redirect the url to: http://$host:11300/result/ # (with '/' at the end) 11300 is the nginx service listen port, so nginx redirect with this port [How] keep both ports are same by set LISTEN_PORT with start script. - set LISTEN_PORT to ENV with start script - use $LISTEN_PORT replace default port(11300) form /etc/nginx/conf.d/default.conf(in container) with container starting Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/sbin/entrypoint.sh | 2 ++ container/srv-http/start-cci | 3 ++- container/srv-http/start-git | 3 ++- container/srv-http/start-initrd | 3 ++- container/srv-http/start-os | 3 ++- container/srv-http/start-pub | 3 ++- container/srv-http/start-result | 3 ++- container/srv-http/start-rpm | 3 ++- 8 files changed, 16 insertions(+), 7 deletions(-) diff --git a/container/srv-http/root/sbin/entrypoint.sh b/container/srv-http/root/sbin/entrypoint.sh index cd73c7e..ced6eda 100755 --- a/container/srv-http/root/sbin/entrypoint.sh +++ b/container/srv-http/root/sbin/entrypoint.sh @@ -1,4 +1,6 @@ #!/bin/sh +sed -i "s/listen 11300;/listen $LISTEN_PORT;/g" /etc/nginx/conf.d/default.conf + nginx /usr/sbin/php-fpm7 diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 00d8e4c..aa539d2 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 20011:11300 + -e LISTEN_PORT=20011 + -p 20011:20011 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-git b/container/srv-http/start-git index c1181a7..0b0ff47 100755 --- a/container/srv-http/start-git +++ b/container/srv-http/start-git @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-git - -p 20010:11300 + -e LISTEN_PORT=20010 + -p 20010:20010 -v /srv/git/archlinux:/srv/git/archlinux:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 466ec1b..93be548 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-initrd - -p 20008:11300 + -e LISTEN_PORT=20008 + -p 20008:20008 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro diff --git a/container/srv-http/start-os b/container/srv-http/start-os index d39e04e..5b63902 100755 --- a/container/srv-http/start-os +++ b/container/srv-http/start-os @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-os - -p 20009:11300 + -e LISTEN_PORT=20009 + -p 20009:20009 -v /srv/os:/srv/os:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub index f42b117..6dfd5a4 100755 --- a/container/srv-http/start-pub +++ b/container/srv-http/start-pub @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-pub - -p 20006:11300 + -e LISTEN_PORT=20006 + -p 20006:20006 -v /srv/pub:/srv/pub:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-result b/container/srv-http/start-result index bf0287d..32bdf7d 100755 --- a/container/srv-http/start-result +++ b/container/srv-http/start-result @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-result - -p 20007:11300 + -e LISTEN_PORT=20007 + -p 20007:20007 -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm index 73e5162..2d38298 100755 --- a/container/srv-http/start-rpm +++ b/container/srv-http/start-rpm @@ -9,7 +9,8 @@ cmd=( docker run --restart=always --name srv-http-rpm - -p 20012:11300 + -e LISTEN_PORT=20012 + -p 20012:20012 -v /srv/rpm/pub:/srv/rpm/pub:ro -v /etc/localtime:/etc/localtime:ro -d -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: update data before get job
by Wu Zhende 06 Apr '21

06 Apr '21

If no job, the session is suspended until the job is obtained. Therefore, testbox message and job events need to be updated before get job to ensure the timeliness. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 11 +++++++++++ src/scheduler/find_job_boot.cr | 16 ++++++++-------- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2e4fd30..f62f83e 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -147,6 +147,17 @@ class Sched @log.warn(e) end + def send_mq_msg(job_state) + mq_msg = { + "job_id" => @env.get?("job_id").to_s, + "testbox" => @env.get?("testbox").to_s, + "deadline" => @env.get?("deadline").to_s, + "time" => get_time, + "job_state" => job_state + } + spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) + end + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 01b6f1d..2fca531 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -25,14 +25,7 @@ class Sched rescue e @log.warn(e) ensure - mq_msg = { - "job_id" => @env.get?("job_id").to_s, - "testbox" => @env.get?("testbox").to_s, - "deadline" => @env.get?("deadline").to_s, - "time" => get_time, - "job_state" => "boot" - } - spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) + send_mq_msg("boot") end # auto submit a job to collect the host information. @@ -196,6 +189,13 @@ class Sched def get_job_boot(host, boot_type) queues = get_queues(host) + + # do before get job from etcd + # because if no job will hang + # need to update information in a timely manner + set_lifecycle(nil, host, queues) + send_mq_msg("boot") + job = get_job_from_queues(queues, host) set_lifecycle(job, host, queues) -- 2.23.0

1 0

[PATCH lkp-tests 2/2] add script for fix dockerfile file
by Bai Jing 06 Apr '21

06 Apr '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/handle_dockerfile.sh | 51 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 lib/handle_dockerfile.sh diff --git a/lib/handle_dockerfile.sh b/lib/handle_dockerfile.sh new file mode 100644 index 000000000..b71f9865d --- /dev/null +++ b/lib/handle_dockerfile.sh @@ -0,0 +1,51 @@ +#!/bin/bash + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$dockerfile_name" +} + +handle_epel() +{ + sed -i "s/epel-release/bash/g" "$dockerfile_name" +} + +handle_epelrpm() +{ + sed -i "s|https://.*epel-release.*rpm|bash|g" "$dockerfile_name" +} + +handle_rpm_gpg() +{ + sed -i "s|RPM-GPG-KEY-CentOS-7|RPM-GPG-KEY-openEuler|g" "$dockerfile_name" + sed -i "s|RPM-GPG-KEY-centosofficial|RPM-GPG-KEY-openEuler|g" "$dockerfile_name" +} + +add_base_commands() +{ + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$dockerfile_name" +} + +handle_epel_repo() +{ + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$dockerfile_name" +} + +handle_error_exit() +{ + sed '/^RUN/RUN set -e;/g' "$dockerfile_name" +} + +handle_dockerfile() +{ + dockerfile_name=$1 + handle_FROM + handle_epel + handle_epelrpm + handle_rpm_gpg + add_base_commands + handle_epel_repo + handle_error_exit +} -- 2.23.0

1 0

[PATCH lkp-tests 1/2] add test script for dockerfile test
by Bai Jing 06 Apr '21

06 Apr '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 143 +++++++++++++++++------------------------ 1 file changed, 58 insertions(+), 85 deletions(-) diff --git a/tests/openeuler_docker b/tests/openeuler_docker index 483f40135..3ea57e2c9 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -8,12 +8,12 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/log.sh +. $LKP_SRC/lib/handle_dockerfile.sh [ -n "$dockerfile_url" ] || die "dockerfile_url is empty" declare -i nr_openeuler_fail=0 declare -i nr_openeuler_success=0 -declare -i is_openeuler_build_fail=0 declare -i dockerfile_count=0 fit_repo() @@ -33,110 +33,51 @@ openeuler_ready() docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar } -centos_docker() -{ - is_openeuler_build_fail=0 - index=1 - for file in $dockerfile_path - do - [[ $file =~ ^centos ]] && continue - file=$(echo "$file" | awk -F: '{print $1}') - dockerfile_name="${file##*/}" - cd "${file%/*}" || continue - image=$(echo "$file" | awk -F '/' '{print $5}' | tr 'A-Z' 'a-z') - [ -n "$image" ] && image=${image}_$index - if [ -f "build.sh" ] && [ -x "build.sh" ]; then - ./build.sh > /dev/null 2> /dev/null - else - docker_build > /dev/null 2> /dev/null - fi - - [ $? == 0 ] && { - handle_dockerfile - } - - let index++ - done -} - -handle_FROM() -{ - sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$dockerfile_name" -} - -handle_epel() -{ - grep -qw "epel-release" "$dockerfile_name" && { - sed -i "s/epel-release/bash/g" "$dockerfile_name" - } -} - -handle_epelrpm() -{ - grep -qw "https://.*epel-release.*rpm" "$dockerfile_name" && { - sed -i "s|https://.*epel-release.*rpm|bash|g" "$dockerfile_name" - } -} - -handle_rpm_gpg() -{ - sed -i "s|RPM-GPG-KEY-CentOS-7|RPM-GPG-KEY-openEuler|g" "$dockerfile_name" -} - -add_base_commands() -{ - # grep -qw "groupadd*" "$file" && { - # fix missing useradd, groupadd, chpasswd, etc. commands - sed -i '/FROM /a\RUN yum -y install shadow tar' "$dockerfile_name" -} - -handle_epel_repo() -{ - sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$dockerfile_name" -} - -handle_dockerfile() +build_centos() { - is_openeuler_build_fail=1 - handle_FROM - handle_epel - handle_rpm_gpg - add_base_commands - if docker_build "-openeuler"; then - docker_run "-openeuler" - let nr_openeuler_success++ + cd ${file%/*} + if [ -f "build.sh" ] && [ -x "build.sh" ]; then + ./build.sh || return 1 else - let nr_openeuler_fail++ + docker build -f "$dockerfile_name" -t "${image}_$1_$index" . || return 1 fi } -docker_build() +build_openeuler() { - docker build -f "$dockerfile_name" -t "${image}$1" . + docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1 [ $? != 0 ] && { - [ "$is_openeuler_build_fail" == "1" ] && echo "${image}$1.build.fail: 1" + let nr_openeuler_fail++ + echo "${image}_$index.build.fail: 1" return 1 } - [ "$is_openeuler_build_fail" == "1" ] && echo "${image}$1.build.success: 1" - is_openeuler_build_fail=0 + echo "${image}_$index.build.success: 1" + let nr_openeuler_success++ + return 0 } docker_run() -{ - docker run -d --name "${image}" "${image}$1" - docker_stats=$(docker inspect -f '{{.State.Running}}' ${image}) - [ $docker_stats == false ] && { - echo "${image}.run.fail: 1" +{ + docker_run_cmd="docker run -d --name ${image}_$1_$index ${image}_$1_$index" + [ $1 == "openeuler" ] && { + docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1" + } + + eval $docker_run_cmd + + docker_stats=$(docker inspect -f '{{.State.Running}}' $image_$1_$index) + [ $docker_stats == "false" ] && { + [ $1 == "openeuler" ] && echo "${image}_$index.run.fail: 1" return 1 } - echo "${image}.run.success: 1" + [ $1 == "openeuler"] && echo "${image}_$index.run.success: 1" } handle_build_result() { if [ "$nr_openeuler_fail" != 0 ]; then - echo "openeuler build failed times: $nr_openeuler_fail" + echo "openeuler_build.nr_failed: $nr_openeuler_fail" echo "openeuler_valid_url: $dockerfile_url" exit 1 # centos all success, openeuler all success @@ -146,6 +87,38 @@ handle_build_result() fi } +build() +{ + dockerfile_name="${file##*/}" + dockerfile_name="${dockerfile_name%:*}" + image=$(echo "$file" | awk -F '/' '{print $5}' | tr 'A-Z' 'a-z') + + build_centos "centos" > /dev/null 2>&1 || return 1 + handle_dockerfile "$dockerfile_name" + build_openeuler "openeuler" + [ $? != 0 ] && return 1 +} + +run() +{ + docker_run "centos" > /dev/null 2>&1 + [ $? != 0 ] && return 1 + docker_run "openeuler" +} + +centos_docker() +{ + index=1 + for file in $dockerfile_path + do + [[ $file =~ ^centos ]] && continue + + build + [ $? = 0 ] && run + let index++ + done +} + docker_test() { fit_repo -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-find: support check query keyword
by Lu Weitao 06 Apr '21

06 Apr '21

[How] 1. add --keyword option 2. use like: es-find --keyword category dequeue_time end_time enqueue_time group_id hw.arch hw.memory hw.model hw.nr_cpu ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 25 +++++++++++++++++++++++++ sbin/es-find | 11 ++++++++++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 928bd68..22d68a9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -130,6 +130,11 @@ class ESQuery parse_fields(result) end + + def query_mapping + mapping = @client.indices.get_mapping(index: @index) + get_mapping_key(mapping) + end end # Range Query Example: @@ -260,3 +265,23 @@ def parse_fields(es_result) result_hash end + +def get_mapping_key(mapping) + mapping_key = [] + properties = mapping.values[0]['mappings']['properties'] + properties.each do |key, value| + parse_mapping_properties(key, value, mapping_key) + end + + mapping_key +end + +def parse_mapping_properties(key, value, mapping_key) + if value['type'] + mapping_key << key + else + value['properties'].each do |k, v| + parse_mapping_properties("#{key}.#{k}", v, mapping_key) + end + end +end diff --git a/sbin/es-find b/sbin/es-find index 2c3b3dc..f548fc1 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -77,6 +77,8 @@ options = { nr_jobs: 10, desc_keyword: 'start_time' } range = nil count = nil +es = ESQuery.new(ES_HOST, ES_PORT) + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -103,6 +105,14 @@ opt_parser = OptionParser.new do |opts| count = c end + opts.on_tail('--keyword', 'check all available search_keys') do + available_search_key = es.query_mapping + available_search_key.each do |key| + puts " #{key}" + end + exit + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -113,7 +123,6 @@ opt_parser.parse!(ARGV) items = parse_argv items[:range] = prase_range(range) if range -es = ESQuery.new(ES_HOST, ES_PORT) if count count_keywords = count.split(',') query_result = es.query_fields(count_keywords, items) -- 2.23.0

1 0

[PATCH v2 compass-ci] delimiter: refactor delimiter with crystal
by Cao Xueliang 04 Apr '21

04 Apr '21

use etcd list watch to submit bisect job Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.cr | 22 ++++++++++ src/delimiter/delimiter.cr | 83 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 src/delimiter.cr create mode 100644 src/delimiter/delimiter.cr diff --git a/src/delimiter.cr b/src/delimiter.cr new file mode 100644 index 0000000..e8de306 --- /dev/null +++ b/src/delimiter.cr @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "yaml" +require "./delimiter/delimiter" + +def config_secrets_yaml + %x(#{ENV["CCI_SRC"]}/sbin/config_account_yaml.rb delimiter) + account = YAML.parse(File.read("#{ENV["HOME"]}/.config/compass-ci/defaults/account.yaml")) + lab = YAML.parse(File.read("#{ENV["HOME"]}/.config/compass-ci/include/lab/#{account["lab"]}.yaml")) + secrets = {"secrets" => lab} + File.open("#{ENV["HOME"]}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + +begin + config_secrets_yaml + delimiter = Delimiter.new + delimiter.consume_delimiter("delimiter") +rescue ex + puts ex +end diff --git a/src/delimiter/delimiter.cr b/src/delimiter/delimiter.cr new file mode 100644 index 0000000..c98ded9 --- /dev/null +++ b/src/delimiter/delimiter.cr @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "../lib/etcd_client" + +class Delimiter + def initialize + @ec = EtcdClient.new + end + + def consume_delimiter(queue) + channel = Channel(Etcd::Model::Kv).new + revision = consume_by_list(queue, channel) + consume_by_watch(queue, revision, channel) + end + + def consume_by_list(queue, channel) + tasks, revision = get_history_tasks(queue) + handle_history_tasks(tasks, channel) + + return revision + end + + def consume_by_watch(queue, revision, channel) + watch_queue(queue, revision, channel) + handle_events(channel) + end + + def get_history_tasks(queue) + tasks = [] of Etcd::Model::Kv + range = @ec.range_prefix(queue) + revision = range.header.not_nil!.revision + tasks += range.kvs + + return tasks, revision + end + + def handle_history_tasks(tasks, channel) + loop do + return if tasks.empty? + + task = tasks.delete_at(0) + spawn { submit_bisect_job(channel, task) } + end + end + + def watch_queue(queue, revision, channel) + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64 + 1, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + events.each do |event| + channel.send(event.kv) + end + end + + spawn { watcher.start } + Fiber.yield + end + + def handle_events(channel) + loop do + task = channel.receive + spawn { submit_bisect_job(channel, task) } + end + end + + def submit_bisect_job(channel, task) + key = task.key + value = Hash(String, String).from_json(task.value.not_nil!) + begin + response = %x(#{ENV["LKP_SRC"]}/sbin/submit bad_job_id=#{value["job_id"]} error_id=#{value["error_id"].inspect} bisect.yaml queue=dc-bisect) + puts response + if /id=0/ =~ response + channel.send(task) + return + end + + @ec.delete(key) + rescue ex + puts ex + channel.send(task) + end + end +end -- 2.23.0

1 0

[PATCH v1 compass-ci 3/3] delimiter: refactor delimiter whit crystal
by Cao Xueliang 04 Apr '21

04 Apr '21

use etcd list watch to submit bisect job Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.cr | 22 ++++++++++ src/delimiter/delimiter.cr | 83 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 src/delimiter.cr create mode 100644 src/delimiter/delimiter.cr diff --git a/src/delimiter.cr b/src/delimiter.cr new file mode 100644 index 0000000..e8de306 --- /dev/null +++ b/src/delimiter.cr @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "yaml" +require "./delimiter/delimiter" + +def config_secrets_yaml + %x(#{ENV["CCI_SRC"]}/sbin/config_account_yaml.rb delimiter) + account = YAML.parse(File.read("#{ENV["HOME"]}/.config/compass-ci/defaults/account.yaml")) + lab = YAML.parse(File.read("#{ENV["HOME"]}/.config/compass-ci/include/lab/#{account["lab"]}.yaml")) + secrets = {"secrets" => lab} + File.open("#{ENV["HOME"]}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + +begin + config_secrets_yaml + delimiter = Delimiter.new + delimiter.consume_delimiter("delimiter") +rescue ex + puts ex +end diff --git a/src/delimiter/delimiter.cr b/src/delimiter/delimiter.cr new file mode 100644 index 0000000..c98ded9 --- /dev/null +++ b/src/delimiter/delimiter.cr @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "../lib/etcd_client" + +class Delimiter + def initialize + @ec = EtcdClient.new + end + + def consume_delimiter(queue) + channel = Channel(Etcd::Model::Kv).new + revision = consume_by_list(queue, channel) + consume_by_watch(queue, revision, channel) + end + + def consume_by_list(queue, channel) + tasks, revision = get_history_tasks(queue) + handle_history_tasks(tasks, channel) + + return revision + end + + def consume_by_watch(queue, revision, channel) + watch_queue(queue, revision, channel) + handle_events(channel) + end + + def get_history_tasks(queue) + tasks = [] of Etcd::Model::Kv + range = @ec.range_prefix(queue) + revision = range.header.not_nil!.revision + tasks += range.kvs + + return tasks, revision + end + + def handle_history_tasks(tasks, channel) + loop do + return if tasks.empty? + + task = tasks.delete_at(0) + spawn { submit_bisect_job(channel, task) } + end + end + + def watch_queue(queue, revision, channel) + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64 + 1, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + events.each do |event| + channel.send(event.kv) + end + end + + spawn { watcher.start } + Fiber.yield + end + + def handle_events(channel) + loop do + task = channel.receive + spawn { submit_bisect_job(channel, task) } + end + end + + def submit_bisect_job(channel, task) + key = task.key + value = Hash(String, String).from_json(task.value.not_nil!) + begin + response = %x(#{ENV["LKP_SRC"]}/sbin/submit bad_job_id=#{value["job_id"]} error_id=#{value["error_id"].inspect} bisect.yaml queue=dc-bisect) + puts response + if /id=0/ =~ response + channel.send(task) + return + end + + @ec.delete(key) + rescue ex + puts ex + channel.send(task) + end + end +end -- 2.23.0

1 0

[PATCH v1 compass-ci 2/3] container/delimiter: compile delimiter with crystal
by Cao Xueliang 04 Apr '21

04 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 11 ++++++++++- container/delimiter/build | 4 ++++ container/delimiter/start | 3 +-- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index cc57320..a9ff279 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.13 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ @@ -11,3 +11,12 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install rest-client activesupport git json yaml elasticsearch faye-websocket terminal-table mail io-console + +ENV RUNTIME_DIR /c/cci/delimiter + +RUN mkdir -p $RUNTIME_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $RUNTIME_DIR + +COPY --chown=1090:1090 delimiter . diff --git a/container/delimiter/build b/container/delimiter/build index 606149d..7f3108b 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -8,6 +8,10 @@ docker_skip_rebuild "delimiter" load_cci_defaults +bash $CCI_SRC/container/compile . + docker build -t delimiter . push_image delimiter:latest + +rm delimiter diff --git a/container/delimiter/start b/container/delimiter/start index 3cff23a..9485750 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -32,10 +32,9 @@ cmd = %w[ -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/register:/etc/compass-ci/register:ro - -w #{DEFAULT_CCI}/src delimiter ] -cmd += ['sh', '-c', 'umask 002 && ruby ./delimiter.rb'] +cmd += ['sh', '-c', 'umask 002 && ./delimiter'] system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 1/3] extract-stats: close etcd client
by Cao Xueliang 04 Apr '21

04 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 11 +++++++---- src/extract-stats/stats_worker.cr | 5 +++-- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 4ae3e71..c35d3ec 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -6,10 +6,13 @@ require "./constants" require "./stats_worker" module ExtractStats + @@ec = EtcdClient.new + def self.consume_tasks channel = Channel(String).new - revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) - self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + queue = EXTRACT_STATS_QUEUE_PATH + revision = self.consume_by_list(queue, channel) + self.consume_by_watch(queue, revision, channel) end def self.consume_by_list(queue, channel) @@ -26,7 +29,7 @@ module ExtractStats def self.get_history_tasks(queue) tasks = [] of Etcd::Model::Kv - range = EtcdClient.new.range_prefix(queue) + range = @@ec.range_prefix(queue) revision = range.header.not_nil!.revision tasks += range.kvs @@ -45,7 +48,7 @@ module ExtractStats end def self.watch_queue(queue, revision, channel) - watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64 + 1, filters: [Etcd::Watch::Filter::NODELETE]) do |events| events.each do |event| puts event channel.send(event.kv.key) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 8910951..45b75a3 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,6 +1,5 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" @@ -30,6 +29,8 @@ class StatsWorker STDERR.puts e.message # incase of many error message when task-queue, ES does not work sleep(10) + ensure + @etcd.close end end @@ -71,7 +72,7 @@ class StatsWorker sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" - value = {"error_id" => sample_error_id} + value = {"job_id" => job_id, "error_id" => sample_error_id} @etcd.put(queue, value) msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: close etcd client
by Cao Xueliang 03 Apr '21

03 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 9 ++++++--- src/extract-stats/stats_worker.cr | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 4ae3e71..3346d12 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -6,10 +6,13 @@ require "./constants" require "./stats_worker" module ExtractStats + @@ec = EtcdClient.new + def self.consume_tasks channel = Channel(String).new - revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) - self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + queue = EXTRACT_STATS_QUEUE_PATH + revision = self.consume_by_list(queue, channel) + self.consume_by_watch(queue, revision, channel) end def self.consume_by_list(queue, channel) @@ -26,7 +29,7 @@ module ExtractStats def self.get_history_tasks(queue) tasks = [] of Etcd::Model::Kv - range = EtcdClient.new.range_prefix(queue) + range = @@ec.range_prefix(queue) revision = range.header.not_nil!.revision tasks += range.kvs diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 8910951..9dea267 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,6 +1,5 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" @@ -30,6 +29,8 @@ class StatsWorker STDERR.puts e.message # incase of many error message when task-queue, ES does not work sleep(10) + ensure + @etcd.close end end -- 2.23.0

1 0

review star
by Wu Fengguang 02 Apr '21

02 Apr '21

Cao Xueliang is our review star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 81 11 11 25 Cao Xueliang 59 12 19 12 Wei Jihui 24 13 28 5 Luan Shengde 22 7 6 3 Cui Yili 19 3 4 3 Wu Zhende 16 3 5 2 Yu Chuan 12 5 5 2 Xu Xijian 2 1 1 1 Wu Fengguang 25 0 0 1 Li Ping 17 1 1 0 Wang Chenglong 20 0 0 0 Liu Yinsi 6 0 0 0 Lu Weitao 1 0 0 0 Zhang Dewan 20 1 1 0 Wang Yong 6 0 0 0 Li Yuanchao 1 0 0 0 Bai Jing 9 4 4 0 Lin Jiaxin 18 5 4 0 Liu Shaofei

1 0