January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/0-package: set umask and selinux
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] some container services need umask 002 to work normally, cci-network.service need disabled or permissive selinux, or will have permission denied questions. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..01c32b2 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -27,6 +27,14 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF +cat >> /etc/bashrc <<EOF +umask 002 +EOF + +source /etc/bashrc + +setenforce 0 + server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

2 1

[PATCH compass-ci 3/4] sparrow/5-build: download rootfs
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] for locally deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03 in deploy script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/sources diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/sources @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index eb9b3cb..a5e314a 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -20,4 +20,5 @@ cd $CCI_SRC/sparrow || exit 4-docker/buildall 5-build/ipxe 5-build/register-accounts +5-build/sources 7-systemd/systemd-setup -- 2.23.0

2 1

[PATCH compass-ci 1/4] sbin: add script for download os rootfs
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] for user deploy compass-ci at their local server, need to download os rootfs to run a qemu kvm or physical machine test. deploy script will call it to download one os rootfs openeuler aarch64 20.03, user can also use this script to download other os rootfs if they want. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100755 index 0000000..5ab427b --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/${os}/${os_arch}/${os_version} + +download_rootfs_file() +{ + mkdir -p /srv/os/${rootfs_dir} + + cd /srv/os/$rootfs_dir || exit + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz + + gzip -dc ${os_version}.cgz |cpio -id + rm -rf ${os_version}.cgz +} + +download_rootfs_file -- 2.23.0

2 1

[PATCH compass-ci 4/4] container: add start-depends for register-accounts container
by Cao Xueliang 27 Jan '21

27 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/register-accounts/start-depends diff --git a/container/register-accounts/start-depends b/container/register-accounts/start-depends new file mode 100755 index 0000000..8357fca --- /dev/null +++ b/container/register-accounts/start-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: add register-accounts container build and Dockerfile
by Cao Xueliang 27 Jan '21

27 Jan '21

This container will register some default accounts which will be used in others service. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/register-accounts/Dockerfile | 15 +++++++++++++++ container/register-accounts/build | 6 ++++++ 2 files changed, 21 insertions(+) create mode 100644 container/register-accounts/Dockerfile create mode 100755 container/register-accounts/build diff --git a/container/register-accounts/Dockerfile b/container/register-accounts/Dockerfile new file mode 100644 index 0000000..933c543 --- /dev/null +++ b/container/register-accounts/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport json yaml elasticsearch io-console diff --git a/container/register-accounts/build b/container/register-accounts/build new file mode 100755 index 0000000..11ac01e --- /dev/null +++ b/container/register-accounts/build @@ -0,0 +1,6 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t register-accounts . + -- 2.23.0

1 0

[PATCH compass-ci 1/4] sparrow: delete register-accounts in install-tiny
by Cao Xueliang 27 Jan '21

27 Jan '21

We will use a container to register accounts. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/install-tiny | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index eb9b3cb..a291549 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,5 +19,4 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe -5-build/register-accounts 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add start depends
by Liu Yinsi 27 Jan '21

27 Jan '21

[why] in Dockerfile and build script, all not depends es, so delete, but in start script, need to query es to get account info, so add start depends es. refer: [PATCH v3 compass-ci] sbin/auto-submit: config account yaml Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 - container/auto-submit/start-depends | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) delete mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends deleted file mode 100644 index 8357fca..0000000 --- a/container/auto-submit/build-depends +++ /dev/null @@ -1 +0,0 @@ -es diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends index 5232abf..38fc696 100755 --- a/container/auto-submit/start-depends +++ b/container/auto-submit/start-depends @@ -1 +1,2 @@ rabbitmq +es -- 2.23.0

1 0

[PATCH lab-z9] add host-file of a7 and a20
by Zhang Dewan 27 Jan '21

27 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a20 | 21 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a7 | 25 +++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a20 create mode 100644 hosts/taishan200-2280-2s64p-256g--a7 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 new file mode 100644 index 0000000..21bf8e1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -0,0 +1,21 @@ +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b888f2dd + - /dev/disk/by-id/scsi-350000399b888f705 + - /dev/disk/by-id/scsi-350000399b88906f5 + - /dev/disk/by-id/scsi-350000399b8890141 +mac_addr: + - 84:46:fe:3d:a7:d5 + - 84:46:fe:3d:a7:d6 + - 84:46:fe:3d:a7:d7 + - 84:46:fe:3d:a7:d8 + - 84:46:fe:3d:a7:d9 + - 84:46:fe:3d:a7:da + - 84:46:fe:3d:a7:db + - 84:46:fe:3d:a7:dc +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 new file mode 100644 index 0000000..07701d3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -0,0 +1,25 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039998134611 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 +mac_addr: + - 84:46:fe:89:be:20 + - 84:46:fe:89:be:21 + - 84:46:fe:89:be:22 + - 84:46:fe:89:be:23 + - ac:b3:b5:1e:d6:eb + - ac:b3:b5:1e:d6:ec + - ac:b3:b5:1e:d6:ed + - ac:b3:b5:1e:d6:ee + - ac:b3:b5:1e:d7:7b + - ac:b3:b5:1e:d7:7c + - ac:b3:b5:1e:d7:7d + - ac:b3:b5:1e:d7:7e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.1.7 -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: update my_info before store into ES
by Liu Yinsi 27 Jan '21

27 Jan '21

before: when store account info to ES, generate new my_token. after: when store account info to ES, if my_email exist, use old my_token, if not exist, generate new my_token. [why] prevent my_token in config yaml is different with ES. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..6aa842c --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -66,12 +66,16 @@ class BuildMyInfo def store_account_info es = ESClient.new(index: 'accounts') + + my_info_es = es.query_by_id(@my_info['my_email']) || {} + @my_info.update my_info_es + es.put_source_by_id(@my_info['my_email'], @my_info) end def config_my_info + store_account_info config_default_yaml config_lab_yaml - store_account_info end end -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/auto-submit: config account yaml
by Liu Yinsi 27 Jan '21

27 Jan '21

before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, my_token from ES, then config yaml [why] store account info into ES, this part will do in other registry service, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..91c659d 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,15 +86,26 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def query_account_info + my_email = 'auto-submit@localhost' + + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +query_account_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 1