- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/4] src/scheduler/kernel_params.cr: add param pv_device into dracut initrd for local mount
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 8ad5029..9e2f58b 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -9,7 +9,8 @@ class Job os_info = "#{os}_#{os_arch}_#{os_version}" use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" if @hash["src_lv_suffix"]? != nil save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" if @hash["boot_lv_suffix"]? != nil - return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw" + pv_device = "#{self.pv_device}" if @hash["pv_device"]? != nil + return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} pv_device=#{pv_device} rw" end private def kernel_custom_params -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/dracut-initrd: enable creating pv and vg for lv to support local mount in HW machine
by Xu Xijian 31 Mar '21

31 Mar '21

[why] If a HW machine has no physical volume or volume group for logical volume, we should support to create them by job param, this param named as pv_device(default empty and do nothing). If a usable $pv_device is given, dracut will check the device and ensure pv and vg. Usage: submit job.yaml ... pv_device=/dev/sda Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 835bdd2..2a07270 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -14,9 +14,36 @@ analyse_kernel_cmdline_params() { sync_src_lv() { local src_lv="$1" + local vg_name="os" [ -e "$src_lv" ] && return + # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda + pv_device="$(getarg pv_device=)" + [ -n "$pv_device" ] && { + [ -b "$pv_device" ] || { + echo "warn dracut: FATAL: device not found: $pv_device, reboot" + reboot + } + + # ensure the physical disk has been initialized as physical volume + real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" + [ "$real_pv_device" = "$pv_device" ] || { + lvm pvcreate "$pv_device" || reboot + } + + # ensure the volume group $vg_name exists + real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" + [ "$real_vg_name" = "$vg_name" ] || { + lvm vgcreate "$vg_name" "$pv_device" || reboot + } + } + + lvm vgs "$vg_name" || { + echo "warn dracut: FATAL: vg os not found, reboot" + reboot + } + # create logical volume src_lv_devname="$(basename $src_lv)" lvm lvcreate -y -L 10G --name "${src_lv_devname#os-}" os -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] container/es: upgrade version
by Wu Zhende 31 Mar '21

31 Mar '21

Upgrade to 7.11.1. Matches the version of kibana so that data can be displayed on kibana. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/es/Dockerfile | 48 ++++++++++++-------------------------- container/es/build | 4 ++-- container/es/start | 2 +- container/logging-es/build | 8 +++---- container/logging-es/start | 2 +- 5 files changed, 23 insertions(+), 41 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index ed02490..44d87ee 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -1,41 +1,24 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 ARG MEMORY -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories - -RUN apk add --no-cache elasticsearch curl - -RUN rm -rf /etc/init.d/elasticsearch \ - && rm -rf /usr/share/java/elasticsearch/plugins \ - && mv /usr/share/java/elasticsearch /usr/share/es \ - && echo "===> Creating Elasticsearch Paths..." \ - && for path in \ - /srv/es \ - /usr/share/es/logs \ - /usr/share/es/config \ - /usr/share/es/config/scripts \ - /usr/share/es/tmp \ - /usr/share/es/plugins \ - ; do \ - mkdir -p "$path"; \ - done \ - && cp /etc/elasticsearch/*.* /usr/share/es/config \ - && chown -R 1090:1090 /usr/share/es \ - && chown -R 1090:1090 /srv/es; - -RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options - -WORKDIR /usr/share/es - -ENV PATH /usr/share/es/bin:$PATH -ENV ES_TMPDIR /usr/share/es/tmp +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options + +RUN mkdir /usr/share/elasticsearch/tmp && \ + chown -R 1090:1090 /usr/share/elasticsearch + +WORKDIR /usr/share/elasticsearch + +ENV PATH /usr/share/elasticsearch/bin:$PATH +ENV ES_TMPDIR /usr/share/elasticsearch/tmp VOLUME ["/srv/es"] @@ -43,4 +26,3 @@ EXPOSE 9200 9300 USER 1090 CMD ["elasticsearch"] - diff --git a/container/es/build b/container/es/build index c7d7115..db5145f 100755 --- a/container/es/build +++ b/container/es/build @@ -5,8 +5,8 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "es643b:alpine311" +docker_skip_rebuild "es:7.11.1" available_memory = get_available_memory -system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." +system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/es/start b/container/es/start index 67d6531..3aa9525 100755 --- a/container/es/start +++ b/container/es/start @@ -15,7 +15,7 @@ cmd=( -v /srv/es:/srv/es -v /etc/localtime:/etc/localtime:ro --name es-server01 - es643b:alpine311 + es:7.11.1 ) "${cmd[@]}" diff --git a/container/logging-es/build b/container/logging-es/build index 3be841a..b50830e 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,15 +5,15 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "logging-es:7.6.2" +docker_skip_rebuild "logging-es:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', - 'x86_64' => 'elasticsearch:7.6.2' + 'aarch64' => 'elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786', + 'x86_64' => 'elasticsearch:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] available_memory = get_available_memory -system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." +system "docker build -t logging-es:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..05ac46f 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -15,7 +15,7 @@ cmd=( -p 9302:9300 -v /srv/es/logging-es:/srv/es/logging-es --name logging-es - logging-es:7.6.2 + logging-es:7.11.1 ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/kibana: upgrade and addition
by Wu Zhende 31 Mar '21

31 Mar '21

1. upgrade from 7.6.2 to 7.11.1 2. add a new kibana for data. one display log and one display data 3. delete logtrail plugin because it don't support in new kibana version Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana-logging/Dockerfile | 12 ++++++++ container/kibana-logging/build | 16 ++++++++++ container/kibana-logging/start | 31 +++++++++++++++++++ container/kibana-logging/start-depends | 1 + container/kibana/Dockerfile | 10 ------- container/kibana/build | 8 ++--- container/kibana/logtrail.json | 41 -------------------------- container/kibana/start | 14 ++++----- container/kibana/start-depends | 2 +- 9 files changed, 72 insertions(+), 63 deletions(-) create mode 100644 container/kibana-logging/Dockerfile create mode 100755 container/kibana-logging/build create mode 100755 container/kibana-logging/start create mode 100755 container/kibana-logging/start-depends delete mode 100644 container/kibana/logtrail.json diff --git a/container/kibana-logging/Dockerfile b/container/kibana-logging/Dockerfile new file mode 100644 index 0000000..970eb5a --- /dev/null +++ b/container/kibana-logging/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} + +# docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml diff --git a/container/kibana-logging/build b/container/kibana-logging/build new file mode 100755 index 0000000..9c75aeb --- /dev/null +++ b/container/kibana-logging/build @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.11.1" + +BASE_IMAGE_DICT = { + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana-logging/start b/container/kibana-logging/start new file mode 100755 index 0000000..f2169a6 --- /dev/null +++ b/container/kibana-logging/start @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' +LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' + +docker_rm 'kibana-logging' + +cmd = %W[ + docker run + --restart=always + --name kibana-logging + -v /etc/localtime:/etc/localtime:ro + -d + -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} + -p 20000:5601 + kibana:7.11.1 +] + +system(*cmd) diff --git a/container/kibana-logging/start-depends b/container/kibana-logging/start-depends new file mode 100755 index 0000000..66c1996 --- /dev/null +++ b/container/kibana-logging/start-depends @@ -0,0 +1 @@ +logging-es diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 60b889d..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -10,13 +10,3 @@ FROM ${BASE_IMAGE} MAINTAINER Wu Zhende <wuzhende666(a)163.com> RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml - -USER root -RUN yum -y install wget \ - && wget https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-7.… -O /logtrail-7.6.2-0.1.31.zip - -USER 1090 -RUN ./bin/kibana-plugin install file:///logtrail-7.6.2-0.1.31.zip \ - && rm -rf /tmp/logtrail-7.6.2-0.1.31.zip - -COPY --chown=1090:1090 logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json diff --git a/container/kibana/build b/container/kibana/build index 9a1fb6d..9c75aeb 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -5,12 +5,12 @@ require_relative '../defconfig' -docker_skip_rebuild "kibana:7.6.2" +docker_skip_rebuild "kibana:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', - 'x86_64' => 'kibana:7.6.2' }.freeze + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json deleted file mode 100644 index c63f2ac..0000000 --- a/container/kibana/logtrail.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "version": 2, - "index_patterns": [{ - "es": { - "default_index": "*", - "allow_url_parameter": false, - "timezone": "CST" - }, - "tail_interval_in_seconds": 10, - "es_index_time_offset_in_seconds": 0, - "display_timezone": "CST", - "display_timestamp_format": "YYYY MMM DD HH:mm:ss", - "max_buckets": 500, - "nested_objects": false, - "default_time_range_in_days": 30, - "max_hosts": 100, - "max_events_to_keep_in_viewer": 10000, - "default_search": "", - "fields": { - "mapping": { - "timestamp": "time", - "display_timestamp" : "time", - "hostname": "container_name", - "program": "tags", - "message": "log" - }, - "message_format": "{{{log}}}", - "keyword_suffix": "keyword" - }, - "color_mapping": { - "field": "level", - "mapping": { - "ERROR": "#FF0000", - "WARN": "#FFEF96", - "DEBUG": "#B5E7A0", - "TRACE": "#CFE0E8", - "INFO": "#339999" - } - } - }] -} diff --git a/container/kibana/start b/container/kibana/start index 26a4944..63cb8f8 100755 --- a/container/kibana/start +++ b/container/kibana/start @@ -7,13 +7,13 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - LOGGING_ES_HOST - LOGGING_ES_PORT + ES_HOST + ES_PORT ] defaults = relevant_defaults(names) -LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' -LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' +ES_HOST = defaults['ES_HOST'] || '172.17.0.1' +ES_PORT = defaults['ES_PORT'] || '9200' docker_rm 'kibana' @@ -23,9 +23,9 @@ cmd = %W[ --name kibana -v /etc/localtime:/etc/localtime:ro -d - -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} - -p 20000:5601 - kibana:7.6.2 + -e ELASTICSEARCH_HOSTS=http://#{ES_HOST}:#{ES_PORT} + -p 20017:5601 + kibana:7.11.1 ] system(*cmd) diff --git a/container/kibana/start-depends b/container/kibana/start-depends index 66c1996..8357fca 100755 --- a/container/kibana/start-depends +++ b/container/kibana/start-depends @@ -1 +1 @@ -logging-es +es -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix no positive? method error
by Li Yuanchao 31 Mar '21

31 Mar '21

The result of query es changed, the result['hits']['total'] changes from a number to a hash, so the positive? method can not be used. Now use the count method which contained by es_client. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index c967c8b..7f36be3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -414,9 +414,9 @@ class MirrorMain new_refs_count: {} } query = { query: { match: { _id: git_repo } } } - result = @es_client.search(index: 'repo', body: query)['hits'] - return fork_stat unless result['total'].positive? + return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? + result = @es_client.search(index: 'repo', body: query)['hits'] fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es: upgrade version
by Wu Zhende 31 Mar '21

31 Mar '21

Upgrade to 7.11.1. Matches the version of kibana so that data can be displayed on kibana. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/es/Dockerfile | 48 ++++++++++++-------------------------- container/es/build | 4 ++-- container/es/start | 2 +- container/logging-es/build | 8 +++---- container/logging-es/start | 2 +- 5 files changed, 23 insertions(+), 41 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index ed02490..44d87ee 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -1,41 +1,24 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 ARG MEMORY -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories - -RUN apk add --no-cache elasticsearch curl - -RUN rm -rf /etc/init.d/elasticsearch \ - && rm -rf /usr/share/java/elasticsearch/plugins \ - && mv /usr/share/java/elasticsearch /usr/share/es \ - && echo "===> Creating Elasticsearch Paths..." \ - && for path in \ - /srv/es \ - /usr/share/es/logs \ - /usr/share/es/config \ - /usr/share/es/config/scripts \ - /usr/share/es/tmp \ - /usr/share/es/plugins \ - ; do \ - mkdir -p "$path"; \ - done \ - && cp /etc/elasticsearch/*.* /usr/share/es/config \ - && chown -R 1090:1090 /usr/share/es \ - && chown -R 1090:1090 /srv/es; - -RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options - -WORKDIR /usr/share/es - -ENV PATH /usr/share/es/bin:$PATH -ENV ES_TMPDIR /usr/share/es/tmp +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options + +RUN mkdir /usr/share/elasticsearch/tmp && \ + chown -R 1090:1090 /usr/share/elasticsearch + +WORKDIR /usr/share/elasticsearch + +ENV PATH /usr/share/elasticsearch/bin:$PATH +ENV ES_TMPDIR /usr/share/elasticsearch/tmp VOLUME ["/srv/es"] @@ -43,4 +26,3 @@ EXPOSE 9200 9300 USER 1090 CMD ["elasticsearch"] - diff --git a/container/es/build b/container/es/build index c7d7115..db5145f 100755 --- a/container/es/build +++ b/container/es/build @@ -5,8 +5,8 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "es643b:alpine311" +docker_skip_rebuild "es:7.11.1" available_memory = get_available_memory -system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." +system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/es/start b/container/es/start index 67d6531..3aa9525 100755 --- a/container/es/start +++ b/container/es/start @@ -15,7 +15,7 @@ cmd=( -v /srv/es:/srv/es -v /etc/localtime:/etc/localtime:ro --name es-server01 - es643b:alpine311 + es:7.11.1 ) "${cmd[@]}" diff --git a/container/logging-es/build b/container/logging-es/build index 3be841a..b50830e 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,15 +5,15 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "logging-es:7.6.2" +docker_skip_rebuild "logging-es:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', - 'x86_64' => 'elasticsearch:7.6.2' + 'aarch64' => 'elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786', + 'x86_64' => 'elasticsearch:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] available_memory = get_available_memory -system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." +system "docker build -t logging-es:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..05ac46f 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -15,7 +15,7 @@ cmd=( -p 9302:9300 -v /srv/es/logging-es:/srv/es/logging-es --name logging-es - logging-es:7.6.2 + logging-es:7.11.1 ) "${cmd[@]}" -- 2.23.0

2 1

[PATCH lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 31 Mar '21

31 Mar '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 102 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..c4a2cd691 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,102 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + next + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test: optimize the output of error_id
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 99 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 83 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..cadc11786 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,107 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 0

[PATCH compass-ci 3/4] doc/job/os_mount.md: add the check pv_device step into document
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index eab95ed..b9b8308 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -47,6 +47,7 @@ The brief flow is as follows: ${boot_lv} -- boot logical volume: - will boot from this lv. - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - check whether ${pv_device} is given, if so, create physical volume(pv for short) and volume group(vg for short) on ${pv_device}. - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. - if ${boot_lv} exists: delete it. - create ${boot_lv} as the snapshot of ${src_lv}. @@ -57,6 +58,7 @@ The brief flow is as follows: ## persistent rootfs data When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. +If you want to run on a brand new machine, you should use 'pv_device' to assign a disk device to create pv and vg. The brief flow is as follows: @@ -65,6 +67,7 @@ The brief flow is as follows: 2. initrd stage: use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} + check and ensure pv and vg exist for lv. - firstly, we need two logical volume: ${src_lv} -- src logical volume: - if have ${use_root_partition}, src_lv=${use_root_partition} @@ -95,6 +98,8 @@ Demo usage: data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 + - if you want to create pv and vg on /dev/sda, you can use like this: + kernel_custom_params: pv_device=/dev/sda ... ``` Notes: @@ -148,6 +153,9 @@ Notes: lvdisplay ${src_lv} > /dev/null && return + # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda + [ -n "$pv_device" ] && { do some check and create pv and vg } + # create logical volume lvcreate --size 10G --name $(basename ${src_lv}) os || exit -- 2.23.0

1 0

[PATCH compass-ci 1/4] src/lib/job.cr: add initrd param pv_device to scheduler
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 5767f61..3ba991d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -90,6 +90,7 @@ class Job linux_vmlinuz_path src_lv_suffix boot_lv_suffix + pv_device ) macro method_missing(call) -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test:
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 99 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 83 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..cadc11786 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,107 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 1

[PATCH lab-z9] hosts: add rootfs_disk field
by Yu Chuan 31 Mar '21

31 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 3 ++- hosts/taishan200-2280-2s48p-256g--a103 | 3 ++- hosts/taishan200-2280-2s48p-256g--a104 | 3 ++- hosts/taishan200-2280-2s48p-256g--a106 | 3 ++- hosts/taishan200-2280-2s48p-256g--a107 | 3 ++- hosts/taishan200-2280-2s48p-256g--a20 | 3 ++- hosts/taishan200-2280-2s48p-256g--a63 | 3 ++- hosts/taishan200-2280-2s48p-256g--a64 | 3 ++- hosts/taishan200-2280-2s48p-256g--a65 | 3 ++- hosts/taishan200-2280-2s48p-512g--a102 | 3 ++- hosts/taishan200-2280-2s48p-512g--a105 | 3 ++- hosts/taishan200-2280-2s48p-512g--a95 | 3 ++- hosts/taishan200-2280-2s64p-256g--a110 | 3 ++- hosts/taishan200-2280-2s64p-256g--a112 | 3 ++- hosts/taishan200-2280-2s64p-256g--a113 | 3 ++- hosts/taishan200-2280-2s64p-256g--a114 | 3 ++- hosts/taishan200-2280-2s64p-256g--a116 | 3 ++- hosts/taishan200-2280-2s64p-256g--a117 | 3 ++- hosts/taishan200-2280-2s64p-256g--a118 | 3 ++- hosts/taishan200-2280-2s64p-256g--a119 | 3 ++- hosts/taishan200-2280-2s64p-256g--a121 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a123 | 3 ++- hosts/taishan200-2280-2s64p-256g--a124 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a5 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a6 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a7 | 4 ++-- 26 files changed, 52 insertions(+), 31 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 index f8a042942dd7..9d40f98e6434 100644 --- a/hosts/taishan200-2280-2s48p-256g--a101 +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -2,11 +2,12 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 256G -nr_hdd_partitions: 3 +nr_hdd_partitions: 2 nr_ssd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b89192e1 - /dev/disk/by-id/scsi-350000399b88bf66d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bef85 ssd_partitions: - /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_BTYF94730FF4480BGN diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 9fcc7187a8a3..ca26e20ec5e4 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 - /dev/disk/by-id/scsi-350000399b88bdd85 - /dev/disk/by-id/scsi-350000399b88bdc4d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bdc7d mac_addr: - 44:67:47:c9:d8:b8 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index 41cdbb052709..d0169640f279 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 - /dev/disk/by-id/scsi-350000399b8892f81 - /dev/disk/by-id/scsi-350000399b8891c49 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88930d1 mac_addr: - 44:67:47:d7:6d:f0 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index 95b2f0500c4c..de0f418250fd 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 - /dev/disk/by-id/scsi-350000399b883ba89 - /dev/disk/by-id/scsi-350000399b883b94d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b883b981 mac_addr: - 44:67:47:86:1c:30 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index 31b8dc6cdf90..f8506b22e38e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 - /dev/disk/by-id/scsi-350000399b8918005 - /dev/disk/by-id/scsi-350000399b8900a4d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8900799 mac_addr: - 84:46:fe:73:a2:71 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 index 21bf8e1fcd9d..6d50eead2f31 100644 --- a/hosts/taishan200-2280-2s48p-256g--a20 +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b888f2dd - /dev/disk/by-id/scsi-350000399b888f705 - /dev/disk/by-id/scsi-350000399b88906f5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8890141 mac_addr: - 84:46:fe:3d:a7:d5 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 index 0b3c938372f3..fea8ccb98892 100644 --- a/hosts/taishan200-2280-2s48p-256g--a63 +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b0431 - /dev/disk/by-id/scsi-350000399886adc71 - /dev/disk/by-id/scsi-350000399886b66f5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b6b75 mac_addr: - 44:67:47:c9:e5:44 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 index 7cd0c4813ac6..b1fc7d2a25d5 100644 --- a/hosts/taishan200-2280-2s48p-256g--a64 +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b2fa1 - /dev/disk/by-id/scsi-350000399b88b4339 - /dev/disk/by-id/scsi-350000399b88b4329 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88aefc1 mac_addr: - 44:67:47:c9:ea:b2 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 index 88441cc536d9..2bafc158a6a0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a65 +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4509 - /dev/disk/by-id/scsi-350000399b88b30dd - /dev/disk/by-id/scsi-350000399b88b4621 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88b4425 mac_addr: - 44:67:47:c9:e9:90 diff --git a/hosts/taishan200-2280-2s48p-512g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 index e1d3f10b8adb..ca88543ffc9b 100644 --- a/hosts/taishan200-2280-2s48p-512g--a102 +++ b/hosts/taishan200-2280-2s48p-512g--a102 @@ -2,10 +2,11 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 nr_ssd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bfbc9 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8919395 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55506 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 66a0aca1f7b0..5a295e906de7 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-512g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d - /dev/disk/by-id/scsi-350000399b8919135 - /dev/disk/by-id/scsi-350000399b8919699 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b89190d5 mac_addr: - 44:67:47:d7:76:c4 diff --git a/hosts/taishan200-2280-2s48p-512g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 index 213d10f5ee77..ab649a3694c9 100644 --- a/hosts/taishan200-2280-2s48p-512g--a95 +++ b/hosts/taishan200-2280-2s48p-512g--a95 @@ -2,11 +2,12 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000039988792e31 - /dev/disk/by-id/scsi-35000039988793705 - /dev/disk/by-id/scsi-35000039988792aa5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399887932fd mac_addr: - cc:05:77:fd:fb:61 diff --git a/hosts/taishan200-2280-2s64p-256g--a110 b/hosts/taishan200-2280-2s64p-256g--a110 index 50625bc632ed..415baa86f1ff 100644 --- a/hosts/taishan200-2280-2s64p-256g--a110 +++ b/hosts/taishan200-2280-2s64p-256g--a110 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6079 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b69bd mac_addr: - 44:67:47:e9:79:3d diff --git a/hosts/taishan200-2280-2s64p-256g--a112 b/hosts/taishan200-2280-2s64p-256g--a112 index 30c0f805c2c4..e53c76b77fe4 100644 --- a/hosts/taishan200-2280-2s64p-256g--a112 +++ b/hosts/taishan200-2280-2s64p-256g--a112 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399888173c1 +rootfs_disk: - /dev/disk/by-id/scsi-350000399888169c9 mac_addr: - 44:a1:91:9d:f3:3a diff --git a/hosts/taishan200-2280-2s64p-256g--a113 b/hosts/taishan200-2280-2s64p-256g--a113 index 826b18d6024d..015daaf0bcab 100644 --- a/hosts/taishan200-2280-2s64p-256g--a113 +++ b/hosts/taishan200-2280-2s64p-256g--a113 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000039988700409 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886bfb25 mac_addr: - 44:67:47:85:ed:b7 diff --git a/hosts/taishan200-2280-2s64p-256g--a114 b/hosts/taishan200-2280-2s64p-256g--a114 index f3a4d68c87c2..448b2e95b640 100644 --- a/hosts/taishan200-2280-2s64p-256g--a114 +++ b/hosts/taishan200-2280-2s64p-256g--a114 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399785bf105 +rootfs_disk: - /dev/disk/by-id/scsi-350000399785bf189 mac_addr: - 44:a1:91:9d:d5:44 diff --git a/hosts/taishan200-2280-2s64p-256g--a116 b/hosts/taishan200-2280-2s64p-256g--a116 index 5ca6cd4c7b2f..4b8dc936fe6d 100644 --- a/hosts/taishan200-2280-2s64p-256g--a116 +++ b/hosts/taishan200-2280-2s64p-256g--a116 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6fad +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b6ba9 mac_addr: - 44:67:47:85:f0:af diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 index 0f104951e8fd..696943cef060 100644 --- a/hosts/taishan200-2280-2s64p-256g--a117 +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b57d1 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b720d mac_addr: - 44:67:47:85:f4:0b diff --git a/hosts/taishan200-2280-2s64p-256g--a118 b/hosts/taishan200-2280-2s64p-256g--a118 index 9a93e034527e..4bcfeea10d66 100644 --- a/hosts/taishan200-2280-2s64p-256g--a118 +++ b/hosts/taishan200-2280-2s64p-256g--a118 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000039988707e91 +rootfs_disk: - /dev/disk/by-id/scsi-350000399887080e5 mac_addr: - 44:67:47:e9:78:cf diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 index 57da86389ada..92d341a5a487 100644 --- a/hosts/taishan200-2280-2s64p-256g--a119 +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6be9 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b5dbd mac_addr: - 44:67:47:e9:79:65 diff --git a/hosts/taishan200-2280-2s64p-256g--a121 b/hosts/taishan200-2280-2s64p-256g--a121 index 9c87315d14c0..94d466828a85 100644 --- a/hosts/taishan200-2280-2s64p-256g--a121 +++ b/hosts/taishan200-2280-2s64p-256g--a121 @@ -1,6 +1,6 @@ memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: +nr_ssd_partitions: 0 +rootfs_disk: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43943 mac_addr: - 44:67:47:e9:76:95 diff --git a/hosts/taishan200-2280-2s64p-256g--a123 b/hosts/taishan200-2280-2s64p-256g--a123 index f7650926e30e..cfc9c58df7b6 100644 --- a/hosts/taishan200-2280-2s64p-256g--a123 +++ b/hosts/taishan200-2280-2s64p-256g--a123 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-3500003998881eb19 +rootfs_disk: - /dev/disk/by-id/scsi-3500003998881e9a1 mac_addr: - 44:a1:91:9d:f5:4a diff --git a/hosts/taishan200-2280-2s64p-256g--a124 b/hosts/taishan200-2280-2s64p-256g--a124 index 7e7bb4222377..975dab549b2b 100644 --- a/hosts/taishan200-2280-2s64p-256g--a124 +++ b/hosts/taishan200-2280-2s64p-256g--a124 @@ -1,6 +1,6 @@ memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: +nr_ssd_partitions: 0 +rootfs_disk: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44344 mac_addr: - 44:67:47:e9:76:9f diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 index e6c11b3def19..37881cce5b20 100644 --- a/hosts/taishan200-2280-2s64p-256g--a5 +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-350000399981306c5 ssd_partitions: - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 diff --git a/hosts/taishan200-2280-2s64p-256g--a6 b/hosts/taishan200-2280-2s64p-256g--a6 index 09441939752c..dc11bb047fc9 100644 --- a/hosts/taishan200-2280-2s64p-256g--a6 +++ b/hosts/taishan200-2280-2s64p-256g--a6 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-350000399981344d9 ssd_partitions: - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000301 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 index 07701d3cf3f9..4329ebfaff1e 100644 --- a/hosts/taishan200-2280-2s64p-256g--a7 +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-35000039998134611 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/kibana: upgrade and addition
by Wu Zhende 31 Mar '21

31 Mar '21

1. upgrade from 7.6.2 to 7.11.1 2. add a new kibana for data. one display log and one display data 3. delete logtrail plugin because it don't support in new kibana version Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana-logging/Dockerfile | 12 ++++++++ container/kibana-logging/build | 16 ++++++++++ container/kibana-logging/start | 31 +++++++++++++++++++ container/kibana-logging/start-depends | 1 + container/kibana/Dockerfile | 10 ------- container/kibana/build | 8 ++--- container/kibana/logtrail.json | 41 -------------------------- container/kibana/start | 14 ++++----- container/kibana/start-depends | 2 +- 9 files changed, 72 insertions(+), 63 deletions(-) create mode 100644 container/kibana-logging/Dockerfile create mode 100755 container/kibana-logging/build create mode 100755 container/kibana-logging/start create mode 100755 container/kibana-logging/start-depends delete mode 100644 container/kibana/logtrail.json diff --git a/container/kibana-logging/Dockerfile b/container/kibana-logging/Dockerfile new file mode 100644 index 0000000..970eb5a --- /dev/null +++ b/container/kibana-logging/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} + +# docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml diff --git a/container/kibana-logging/build b/container/kibana-logging/build new file mode 100755 index 0000000..9c75aeb --- /dev/null +++ b/container/kibana-logging/build @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.11.1" + +BASE_IMAGE_DICT = { + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana-logging/start b/container/kibana-logging/start new file mode 100755 index 0000000..f2169a6 --- /dev/null +++ b/container/kibana-logging/start @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' +LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' + +docker_rm 'kibana-logging' + +cmd = %W[ + docker run + --restart=always + --name kibana-logging + -v /etc/localtime:/etc/localtime:ro + -d + -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} + -p 20000:5601 + kibana:7.11.1 +] + +system(*cmd) diff --git a/container/kibana-logging/start-depends b/container/kibana-logging/start-depends new file mode 100755 index 0000000..66c1996 --- /dev/null +++ b/container/kibana-logging/start-depends @@ -0,0 +1 @@ +logging-es diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 60b889d..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -10,13 +10,3 @@ FROM ${BASE_IMAGE} MAINTAINER Wu Zhende <wuzhende666(a)163.com> RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml - -USER root -RUN yum -y install wget \ - && wget https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-7.… -O /logtrail-7.6.2-0.1.31.zip - -USER 1090 -RUN ./bin/kibana-plugin install file:///logtrail-7.6.2-0.1.31.zip \ - && rm -rf /tmp/logtrail-7.6.2-0.1.31.zip - -COPY --chown=1090:1090 logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json diff --git a/container/kibana/build b/container/kibana/build index 9a1fb6d..9c75aeb 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -5,12 +5,12 @@ require_relative '../defconfig' -docker_skip_rebuild "kibana:7.6.2" +docker_skip_rebuild "kibana:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', - 'x86_64' => 'kibana:7.6.2' }.freeze + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json deleted file mode 100644 index c63f2ac..0000000 --- a/container/kibana/logtrail.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "version": 2, - "index_patterns": [{ - "es": { - "default_index": "*", - "allow_url_parameter": false, - "timezone": "CST" - }, - "tail_interval_in_seconds": 10, - "es_index_time_offset_in_seconds": 0, - "display_timezone": "CST", - "display_timestamp_format": "YYYY MMM DD HH:mm:ss", - "max_buckets": 500, - "nested_objects": false, - "default_time_range_in_days": 30, - "max_hosts": 100, - "max_events_to_keep_in_viewer": 10000, - "default_search": "", - "fields": { - "mapping": { - "timestamp": "time", - "display_timestamp" : "time", - "hostname": "container_name", - "program": "tags", - "message": "log" - }, - "message_format": "{{{log}}}", - "keyword_suffix": "keyword" - }, - "color_mapping": { - "field": "level", - "mapping": { - "ERROR": "#FF0000", - "WARN": "#FFEF96", - "DEBUG": "#B5E7A0", - "TRACE": "#CFE0E8", - "INFO": "#339999" - } - } - }] -} diff --git a/container/kibana/start b/container/kibana/start index 26a4944..ac74ecc 100755 --- a/container/kibana/start +++ b/container/kibana/start @@ -7,13 +7,13 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - LOGGING_ES_HOST - LOGGING_ES_PORT + ES_HOST + ES_PORT ] defaults = relevant_defaults(names) -LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' -LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' +ES_HOST = defaults['ES_HOST'] || '172.17.0.1' +ES_PORT = defaults['ES_PORT'] || '9200' docker_rm 'kibana' @@ -23,9 +23,9 @@ cmd = %W[ --name kibana -v /etc/localtime:/etc/localtime:ro -d - -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} - -p 20000:5601 - kibana:7.6.2 + -e ELASTICSEARCH_HOSTS=http://#{ES_HOST}:#{ES_PORT} + -p 20014:5601 + kibana:7.11.1 ] system(*cmd) diff --git a/container/kibana/start-depends b/container/kibana/start-depends index 66c1996..8357fca 100755 --- a/container/kibana/start-depends +++ b/container/kibana/start-depends @@ -1 +1 @@ -logging-es +es -- 2.23.0

1 0

[PATCH compass-ci] lib/stats_wrapper.rb: warn if stats-script doesn't exists
by Lu Weitao 31 Mar '21

31 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index faa81df..d7ef9ce 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -29,6 +29,11 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" + unless File.exist?("#{PROGRAM_DIR}/#{@program}") + warn "#{PROGRAM_DIR}/#{@program} doesn't exist" + return + end + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile -- 2.23.0

1 0

Re: [PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Li Ping 31 Mar '21

31 Mar '21

rename var from vm_name to docker_name ---------^ Thanks, Ping Reply-To: In-Reply-To: <20210330074124.1780870-1-weijihuiall(a)163.com> On Tue, Mar 30, 2021 at 03:41:24PM +0800, Wei Jihui wrote: >vm_name --> docker_name > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > container/ssh-r/start | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > >diff --git a/container/ssh-r/start b/container/ssh-r/start >index 68a63f2..47a29eb 100755 >--- a/container/ssh-r/start >+++ b/container/ssh-r/start >@@ -4,14 +4,14 @@ > > . $CCI_SRC/container/defconfig.sh > >-vm_name=ssh_r >+docker_name=ssh_r > >-docker_rm ${vm_name} >+docker_rm ${docker_name} > > cmd=( > docker run > --restart=always >- --name ${vm_name} >+ --name ${docker_name} > -e GATEWAY_PORTS=true > -e TCP_FORWARDING=true > -d >-- >2.23.0 >

1 0

[PATCH compass-ci] providers/docker: replace URI.open to %x(curl url)
by Cao Xueliang 31 Mar '21

31 Mar '21

curl can suspend when the response need more time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/docker.rb | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 51017f3..db0ccee 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -39,15 +39,10 @@ def del_host2queues(hostname) end def parse_response(url) - response = nil - URI.open(url) do |http| - response = http.read - end + response = %x(curl #{url}) hash = response.is_a?(String) ? JSON.parse(response) : nil if hash.nil? || !hash.key?('job') - puts '..........' - puts 'no job now' - puts '..........' + puts response return nil end return hash -- 2.23.0

1 0

[PATCH compass-ci] docker/run.sh: do not forward logs
by Wu Zhende 31 Mar '21

31 Mar '21

After the configuration, docker logs are dumped tp fluentd. The temporary docker also generates an index in logging-es. This is unnecessary and wastes es resources. Save temporary docker logs to the local host. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 80950dd..516ebbc 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -36,6 +36,8 @@ cmd=( -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -v ${busybox_path}:/usr/local/bin/busybox + --log-driver json-file + --log-opt max-size=10m --oom-score-adj="-1000" ${docker_image} /root/sbin/entrypoint.sh -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test:
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 104 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 87 insertions(+), 17 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..0ede72c8f 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,110 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end -while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + #when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end - ansible_failed_json = JSON.parse @ansible_failed_info +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case ansible_failed_json['failures'] + when /.*/ + p '1' + else + output_error ansible_failed_json['failures'] + end + end +end + +while (line = STDIN.gets) + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + puts "module_name: #{@module_name['action']}" + when /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 1

[PATCH v2 compass-ci 2/2] src/monitor: detect crash and send msg to mq queue.
by Li Ping 30 Mar '21

30 Mar '21

linux/kernel/panic.c: pr_emerg("Kernel panic - not syncing: %s\n", buf); When kernel panic happens, it will print the panic call stack, then the ending msg like: [ 14.405869] Kernel panic - not syncing: Fatal exception in interrupt some bug not cause kernel panic happens but also mean that job fails like: mount.nfs: Connection timed out No space left on device The above message would be detected, then send msg to mq queue to close job and reboot machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 47 +++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 451312c..ff3b84a 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,16 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + "Kernel panic - not syncing: ", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -51,7 +59,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, pattern_list) + def detect_patterns(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,17 +68,50 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, signal) + boundary_signal = detect_patterns(msg, signal) return unless boundary_signal @host2head.delete(host) @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_patterns(msg, CRASH_PATTERNS) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] src/lib: add on_job_crash
by Li Ping 30 Mar '21

30 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index adcb965..0e6cb82 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -189,6 +191,16 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + close_job(event_job_id, "crash") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -296,6 +308,15 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + reboot_machine(testbox, machine) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) @@ -308,6 +329,10 @@ class Lifecycle deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) return if Time.local < deadline + reboot_machine(testbox, machine) + end + + def reboot_machine(testbox, machine) mq_queue = get_machine_reboot_queue(testbox) machine.as_h.delete("history") machine.as_h["testbox"] = JSON::Any.new(testbox) -- 2.23.0

1 0

[PATCH compass-ci] etcd/Dockerfile: change source address
by Wang Yong 30 Mar '21

30 Mar '21

accelerate download Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/etcd/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index 3654fc4..b3957e8 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ARG VERSION RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && apk add ca-certificates && \ - wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-arm64.ta… && \ tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* -- 2.23.0

1 0

[PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] container/taskqueue: compiling code every time
by Wu Zhende 30 Mar '21

30 Mar '21

Delete the complied taskqueue file. Otherwise, the code will be updated and the taskqueue file will not be recomplied. Uses old code all the time. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/taskqueue/build | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/container/taskqueue/build b/container/taskqueue/build index 4f69f85..518a27e 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -12,10 +12,8 @@ service_port=${TASKQUEUE_PORT:-3060} docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" -if [[ ! -f "taskqueue" ]]; then - $CCI_SRC/container/compile . -else - echo -e "\n\033[34m- Info: using \033[31mexists\033[34m taskqueue -\033[0m\n" -fi +$CCI_SRC/container/compile . docker build -t taskqueue-alpine:v0.001-${service_port} . + +rm taskqueue -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix useless arg
by Wei Jihui 30 Mar '21

30 Mar '21

host is not used at detect_start_or_end Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/monitoring/parse_serial_logs.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..451312c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -51,7 +51,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, host, pattern_list) + def detect_start_or_end(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,7 +60,7 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, host, signal) + boundary_signal = detect_start_or_end(msg, signal) return unless boundary_signal @host2head.delete(host) -- 2.23.0

1 0

[PATCH lkp-tests] monitors/kmsg: not execute dmesg when in docker
by Lin Jiaxin 30 Mar '21

30 Mar '21

In order to avoid the following stderr: ==> /tmp/stderr <== dmesg: read kernel buffer failed: Operation not permitted Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- monitors/kmsg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/monitors/kmsg b/monitors/kmsg index 4edd90a9..a1cf7121 100755 --- a/monitors/kmsg +++ b/monitors/kmsg @@ -6,6 +6,8 @@ # serial console has better availability. is_virt && [ -z "$LKP_LOCAL_RUN" ] && exec > /dev/ttyS1 +[ -f /.dockerenv ] && exit 0 + if dmesg --help 2>&1 | grep -q -- --follow; then exec dmesg --follow --decode else -- 2.23.0

2 1