- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/4] container/updaterepo: add a new docker service
by Li Ping 07 Apr '21

07 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ container/updaterepo/build | 14 ++++++++++++++ container/updaterepo/start | 24 ++++++++++++++++++++++++ 3 files changed, 59 insertions(+) create mode 100644 container/updaterepo/Dockerfile create mode 100755 container/updaterepo/build create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..4ac1925 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 10003:10003 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci] sched: fix split jobs by email
by Cao Xueliang 07 Apr '21

07 Apr '21

The queue is "/queues/sched/$queue/ready/$subqueue/$id", $subqueue is the email, so we should get the 5 index when split by "/". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 99c577f..4f49394 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -117,7 +117,7 @@ class Sched def split_jobs_by_email(jobs) hash = Hash(String, Array(Etcd::Model::Kv)).new jobs.each do |job| - key = job.key.split("/")[4] + key = job.key.split("/")[5] if hash.has_key?(key) hash[key] << job else -- 2.23.0

1 0

[PATCH v1 compass-ci] sparrow: etcd-server no need to deploy if it's running
by Cao Xueliang 07 Apr '21

07 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/4-docker/buildall | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3611158..e228153 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -58,10 +58,15 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == "ssh-r" ] && { - container_id=$(docker ps -aqf name="ssh_r") - [ -n "$container_id" ] && exit - } + declare -A dic + local dic=(['ssh-r']='ssh_r' ['etcd']='etcd-server') + for item in ${!dic[@]} + do + [ "$container_name" == ${item} ] && { + container_id=$(docker ps -aqf name=${dic[${item}]}) + [ -n "$container_id" ] && exit + } + done [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci] sparrow: etcd-server no need to deploy if it's running
by Cao Xueliang 07 Apr '21

07 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/4-docker/buildall | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3611158..532f2b4 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -58,10 +58,15 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == "ssh-r" ] && { - container_id=$(docker ps -aqf name="ssh_r") - [ -n "$container_id" ] && exit - } + declare -A dic + local dic=(['ssh-r']='ssh_r' ['etcd']='etcd-server') + for item in ${!dic[@]} + do + [ "$container_name" == ${item} ] && { + container_id=$(docker ps -aqf name=${dic[${item}}) + [ -n "$container_id" ] && exit + } + done [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run -- 2.23.0

1 1

[PATCH lab-z9] add some host files
by Zhang Dewan 06 Apr '21

06 Apr '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a16 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a87 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a99 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a122 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a125 | 23 +++++++++++++++++++++++ 5 files changed, 105 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a16 create mode 100644 hosts/taishan200-2280-2s48p-256g--a87 create mode 100644 hosts/taishan200-2280-2s48p-256g--a99 create mode 100644 hosts/taishan200-2280-2s64p-256g--a122 create mode 100644 hosts/taishan200-2280-2s64p-256g--a125 diff --git a/hosts/taishan200-2280-2s48p-256g--a16 b/hosts/taishan200-2280-2s48p-256g--a16 new file mode 100644 index 0000000..7aba0af --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a16 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bf271 + - /dev/disk/by-id/scsi-350000399b88bf1a9 + - /dev/disk/by-id/scsi-350000399b88bf0d5 + - /dev/disk/by-id/scsi-350000399b88bf385 +mac_addr: + - 84:46:fe:3d:a2:17 + - 84:46:fe:3d:a2:18 + - 84:46:fe:3d:a2:19 + - 84:46:fe:3d:a2:1a + - 84:46:fe:3d:a2:1b + - 84:46:fe:3d:a2:1c + - 84:46:fe:3d:a2:1d + - 84:46:fe:3d:a2:1e +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a87 b/hosts/taishan200-2280-2s48p-256g--a87 new file mode 100644 index 0000000..db88e7e --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a87 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a6b55 + - /dev/disk/by-id/scsi-350000399886ad4a5 + - /dev/disk/by-id/scsi-350000399886a7139 + - /dev/disk/by-id/scsi-350000399886acb21 +mac_addr: + - 44:67:47:c9:eb:66 + - 44:67:47:c9:eb:67 + - 44:67:47:c9:eb:68 + - 44:67:47:c9:eb:69 + - 44:67:47:c9:eb:6a + - 44:67:47:c9:eb:6b + - 44:67:47:c9:eb:6c + - 44:67:47:c9:eb:6d +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.8.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a99 b/hosts/taishan200-2280-2s48p-256g--a99 new file mode 100644 index 0000000..205b5cd --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a99 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b5f81 + - /dev/disk/by-id/scsi-350000399886a7199 + - /dev/disk/by-id/scsi-350000399886a6ea9 + - /dev/disk/by-id/scsi-35000c500bd66d04b +mac_addr: + - 5c:e8:83:73:16:43 + - 5c:e8:83:73:16:44 + - 5c:e8:83:73:16:45 + - 5c:e8:83:73:16:46 + - 5c:e8:83:73:16:47 + - 5c:e8:83:73:16:48 + - 5c:e8:83:73:16:49 + - 5c:e8:83:73:16:4a +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.4 diff --git a/hosts/taishan200-2280-2s64p-256g--a122 b/hosts/taishan200-2280-2s64p-256g--a122 new file mode 100644 index 0000000..5e3d9b7 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a122 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-3500003998881eb0d + - /dev/disk/by-id/scsi-35000039988816c75 +mac_addr: + - 44:a1:91:9d:d8:25 + - 44:a1:91:9d:d8:26 + - 44:a1:91:9d:d8:27 + - 44:a1:91:9d:d8:28 + - 44:a1:91:e5:7b:01 + - 44:a1:91:e5:7b:02 + - 44:a1:91:e5:7b:03 + - 44:a1:91:e5:7b:04 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.3 diff --git a/hosts/taishan200-2280-2s64p-256g--a125 b/hosts/taishan200-2280-2s64p-256g--a125 new file mode 100644 index 0000000..5c5862e --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a125 @@ -0,0 +1,23 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988707ca1 + - scsi-35000039988707ca10 + - scsi-35000039988707ca1-part* + - /dev/disk/by-id/scsi-35000039988707d25-part1 + - /dev/disk/by-id/scsi-35000039988707d25-part2 + - /dev/disk/by-id/scsi-35000039988707d25-part3 +mac_addr: + - 44:67:47:85:f4:65 + - 44:67:47:85:f4:66 + - 44:67:47:85:f4:67 + - 44:67:47:85:f4:68 + - 64:2c:ac:df:06:50 + - 64:2c:ac:df:06:51 + - 64:2c:ac:df:06:52 + - 64:2c:ac:df:06:53 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.6 -- 2.23.0

2 1

[PATCH compass-ci 4/4] add update-repo service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/web_env.cr | 9 +++++++++ src/updaterepo.cr | 17 +++++++++++++++++ src/updaterepo/constants.cr | 4 ++++ src/updaterepo/updaterepo.cr | 16 ++++++++++++++++ 4 files changed, 46 insertions(+) create mode 100644 src/updaterepo.cr create mode 100644 src/updaterepo/constants.cr create mode 100644 src/updaterepo/updaterepo.cr diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index b3a90a8..d47e7e5 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -4,6 +4,7 @@ require "./sched" require "./lifecycle" require "./json_logger" +require "./updaterepo" class HTTP::Server # Instances of this class are passed to an `HTTP::Server` handler. @@ -31,5 +32,13 @@ class HTTP::Server def create_lifecycle @lifecycle = Lifecycle.new(self) end + + def repo + @repo ||= create_repo + end + + def create_repo + @repo = Repo.new(self) + end end end diff --git a/src/updaterepo.cr b/src/updaterepo.cr new file mode 100644 index 0000000..05fa4fe --- /dev/null +++ b/src/updaterepo.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "./updaterepo/updaterepo" +require "./updaterepo/constants" +require "./lib/json_logger" + + +module Updaterepo + log = JSONLogger.new + + begin + Kemal.run(REPO_PORT) + rescue e + log.error(e) + end +end diff --git a/src/updaterepo/constants.cr b/src/updaterepo/constants.cr new file mode 100644 index 0000000..25d9978 --- /dev/null +++ b/src/updaterepo/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +REPO_PORT = (ENV.has_key?("REPO_PORT") ? ENV["REPO_PORT"] : 10003).to_i32 diff --git a/src/updaterepo/updaterepo.cr b/src/updaterepo/updaterepo.cr new file mode 100644 index 0000000..7ad9d20 --- /dev/null +++ b/src/updaterepo/updaterepo.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/updaterepo" +require "../lib/json_logger" + +module Updaterepo + VERSION = "0.1.0" + post "/upload" do |env| + env.repo.upload_repo + "done" + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] src/lib/updaterepo.cr: send messages to update-repo queue
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/updaterepo.cr | 54 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 src/lib/updaterepo.cr diff --git a/src/lib/updaterepo.cr b/src/lib/updaterepo.cr new file mode 100644 index 0000000..3587de5 --- /dev/null +++ b/src/lib/updaterepo.cr @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "json" +require "kemal" +require "./json_logger" +require "./mq" + +class Repo + def initialize(env : HTTP::Server::Context) + @mq = MQClient.instance + @env = env + @log = JSONLogger.new + end + + def upload_repo + begin + body = @env.request.body.not_nil!.gets_to_end + data = JSON.parse(body.to_s).as_h? + rescue e + @log.error(e) + end + + begin + check_params_complete(data) + mq_publish(data) + rescue e + response = { "errcode" => "101", "errmsg" => "upload rpm failed" } + @log.info(response.to_json) + puts "error message: #{e.message}" + end + end + + def mq_publish(data) + mq_msg = data + spawn mq_publish_check("update_repo", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def check_params_complete(params) + params = params.not_nil! + tmp_hash = {"errcode" => "101", "errmsg" => "no upload_rpms params"}.to_json + raise tmp_hash unless params["upload_rpms"]? + end +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/updaterepo: add a new docker service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ container/updaterepo/build | 14 ++++++++++++++ container/updaterepo/start | 24 ++++++++++++++++++++++++ 3 files changed, 59 insertions(+) create mode 100644 container/updaterepo/Dockerfile create mode 100755 container/updaterepo/build create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..4ac1925 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 10003:10003 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/rpm-repo: add a new docker service
by Li Ping 06 Apr '21

06 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/Dockerfile | 10 ++++++ container/rpm-repo/build | 6 ++++ container/rpm-repo/start | 41 ++++++++++++++++++++++ lib/rpm-repo.rb | 65 +++++++++++++++++++++++++++++++++++ 4 files changed, 122 insertions(+) create mode 100644 container/rpm-repo/Dockerfile create mode 100755 container/rpm-repo/build create mode 100755 container/rpm-repo/start create mode 100755 lib/rpm-repo.rb diff --git a/container/rpm-repo/Dockerfile b/container/rpm-repo/Dockerfile new file mode 100644 index 0000000..588828a --- /dev/null +++ b/container/rpm-repo/Dockerfile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fedora:latest + +RUN yum install -y createrepo ruby + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install bunny diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . diff --git a/container/rpm-repo/start b/container/rpm-repo/start new file mode 100755 index 0000000..08fa434 --- /dev/null +++ b/container/rpm-repo/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +DOCKER_CCI = '/c/compass-ci' + +names = Set.new %w[ + MQ_HOST + MQ_PORT + LKP_SERVER +] + +defaults = relevant_defaults(names) +defaults['MQ_HOST'] ||= defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['MQ_PORT'] ||= 5672 +env = docker_env(defaults) + +docker_rm "update-repodata" + +cmd = %W[ + docker run + --restart=always + --name update-repodata + -u 1090:1090 +] + env + %W[ + -d + -e CCI_SRC=#{DOCKER_CCI} + -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} + -v /etc/localtime:/etc/localtime:ro + -v /srv/rpm:/srv/rpm + -w /c/compass-ci/lib + fedora:update-repodata +] + +cmd += ['sh', '-c', 'umask 002 && ./rpm-repo.rb'] +system(*cmd) diff --git a/lib/rpm-repo.rb b/lib/rpm-repo.rb new file mode 100755 index 0000000..f516d3f --- /dev/null +++ b/lib/rpm-repo.rb @@ -0,0 +1,65 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'fileutils' +require 'json' +require './mq_client' + +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || '172.17.0.1' +MQ_PORT = ENV['MQ_PORT'] || 5672 + +# used to handle the mq queue "update_repo" +# +# Example items in @mq.queue "update_repo": { "upload_rpms" => ["/srv/rpm/upload/**/Packages/*.rpm", "/srv/rpm/upload/**/source/*.rpm"]} +# handle_new_rpm +# move /srv/rpm/upload/**/*.rpm to /srv/rpm/testing/**/*.rpm +# update /srv/rpm/testing/**/repodate +# update_pub_dir +# copy /srv/rpm/testing/**/*.rpm to /srv/rpm/pub/**/*.rpm +# change /srv/rpm/pub/**/repodate +# @mq.ack(info) +class HandleRepo + def initialize + @mq = MQClient.new(MQ_HOST, MQ_PORT) + @update = [] + end + + def handle_new_rpm + queue = @mq.queue("update_repo") + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + rpm_info = JSON.parse(msg) + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + + dest = File.join(rpm_path.to_s, File.basename(rpm)) + @update << dest + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + update_pub_dir + @mq.ack(info) + end + end + + def update_pub_dir + @update.each do |rpm| + pub_path = File.dirname(rpm).sub("testing", "pub") + FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) + + dest = File.join(pub_path, File.basename(rpm)) + FileUtils.cp(rpm, dest) + + repodata_dest = File.join(File.dirname(pub_path), "repodata") + repodata_src = File.dirname(rpm).sub("Packages", "repodata") + + FileUtils.rm_r(repodata_dest) if Dir.exist?(repodata_dest) + FileUtils.cp_r(repodata_src, File.dirname(repodata_dest)) + end + end +end + +hr = HandleRepo.new +hr.handle_new_rpm -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add build-pkg.rb to generate error_id with message
by Lin Jiaxin 06 Apr '21

06 Apr '21

output like this: "build-pkg.include/common/ist.h:warning:unused-parameter‘count’[-Wunused-parameter]": 1, "build-pkg.include/common/ist.h:warning:unused-parameter‘count’[-Wunused-parameter].message": [ "include/common/ist.h:488:74: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str(char *dst, const struct ist src, size_t count)\n ^~~~~\n", "include/common/ist.h:533:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_lc(char *dst, const struct ist src, size_t count )\n ^~~~~\n", "include/common/ist.h:578:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_uc(char *dst, const struct ist src, size_t count )\n ^~~~~\n" ] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg.rb | 74 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 stats/build-pkg.rb diff --git a/stats/build-pkg.rb b/stats/build-pkg.rb new file mode 100755 index 00000000..4515bf23 --- /dev/null +++ b/stats/build-pkg.rb @@ -0,0 +1,74 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require 'set' +require "#{LKP_SRC}/lib/common" + +class ErrorMessages + def initialize + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages(log_lines) + log_lines.each do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end +end + +def build_pkg(log_lines) + message = {} + ErrorMessages.new.obtain_error_messages(log_lines).each do |k, v| + message[k] = '1' + message["#{k}.message"] = v.to_a + end + + return message +end -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci