- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/build-pkg: matching url is not printed in function replace_source
by Lin Jiaxin 15 Apr '21

15 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 350d8fdf..b399fdc3 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -122,7 +122,7 @@ replace_source() echo "source=(" > $1 for url in ${source[@]} do - echo "$url" | egrep 'git\+|\.git$' && { + echo "$url" | egrep 'git\+|\.git$' > /dev/null && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 15 Apr '21

15 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg.yaml" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

2 2

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 14 Apr '21

14 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

1 0

[PATCH lkp-tests] tests: script to deploy ceph for multi server test
by Luan Shengde 14 Apr '21

14 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/ceph-init | 152 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100755 tests/ceph-init diff --git a/tests/ceph-init b/tests/ceph-init new file mode 100755 index 000000000..735376df6 --- /dev/null +++ b/tests/ceph-init @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# - cephuser_password +# - cephuser + +set -x + +: "${cephuser:=cephuser}" +: "${cephuser_password:=cephuser_password}" + +### ===== on each node ===== +# stop firewalld and disable selinx +systemctl stop firewalld +systemctl disable firewalld + +sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux + +until [[ -n $node_roles ]]; do + sleep 1 +done + +hosts=(cephnode1 cephnode2 cephnode3) + +[[ $node_roles == cephnode1 ]] && { + hostnamectl set-hostname cephnode1 +} + +[[ $node_roles == cephnode2 ]] && { + hostnamectl set-hostname cephnode2 +} + +[[ $node_roles == cephnode3 ]] && { + hostnamectl set-hostname cephnode3 +} + +yum update +yum install ntp wget gcc make ceph lvm2 python3 -y + +# add user and set sudo permissions +useradd -d /home/$cephuser -m $cephuser +echo $cephuser:$cephuser_password | chpasswd + +cat >> /etc/sudoers <<-EOF +$cephuser ALL=(ALL) NOPASSWD:ALL +EOF + +# check and install pip +[[ $(command -v pip) ]] || { + until wget https://bootstrap.pypa.io/get-pip.py; do + sleep 2 + done + python get-pip.py +} + +pip install prettytable + +### ===== on ceph-deploy node ===== +# get node IPs is first +[[ $node_roles == cephnode1 ]] || exit 0 + +until [[ -n $cephnode1 ]]; do + sleep 1 +done + +until [[ -n $cephnode2 ]]; do + sleep 1 +done + +until [[ -n $cephnode3 ]]; do + sleep 1 +done + +# config hosts and set default ssh users +cat >> /etc/hosts <<-EOF +$cephnode1 cephnode1 cephnode1 +$cephnode2 cephnode2 cephnode2 +$cephnode3 cephnode3 cephnode3 +EOF + +for host in "${hosts[@]}"; do + cat >> "$HOME/.ssh/config" <<-EOF + Host $host + Hostname $host + User $cephuser + EOF +done + +chmod 0600 "$HOME/.ssh/config" + + +network=$(ip route | grep default | awk '{print $3}' | awk -F'.' '{print $1"."$2}') +net_mask=$(ip route | grep "^$network" | awk '{print $1}') + +# do the deploy until all the servers are up +# for host in "${host[@]}"; do +# until ping -c 1 -W 1 172.168.131.100> /dev/null; do +# sleep 2 +# done +# done + +# set server trust +[[ $(command -v sshpass) ]] || { + until wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz; do + sleep 2 + done +} + +tar xzvf sshpass.tar.gz && { + cd sshpass-1.08 || exit 1 + ./configure + make + make install +} + +rm -rf "$HOME/.ssh/id_rsa" "$HOME/.ssh/id_rsa.pub" +ssh-keygen -f "$HOME/.ssh/id_rsa" -t rsa -N '' + +sed -i '/StrictHostKeyChecking/c StrictHostKeyChecking no' /etc/ssh/ssh_config +for host in "${hosts[@]}"; do +{ + sshpass -p $cephuser_password ssh-copy-id "$cephuser"@"$host" &> /dev/null +}& +done +wait + +# install +pip install ceph-deploy + +pip_dir=$(pip show ceph-deploy | awk '/^Location/ {print $NF}') +ceph_init_file="${pip_dir}/ceph_deploy/hosts/__init__.py" +[[ -f $ceph_init_file ]]&& { + sed -i "/distributions = {/a 'openeuler': fedora," "$ceph_init_file" +} + +# deploy ceph cluster +disk_name=$(sed -e '1d' -e '/^$/d' /proc/partitions | grep " .da" | awk '{print $NF}') +disk_device="/dev/${disk_name}" + +mkdir -p "$HOME/ceph-deploy" +cd "$HOME/ceph-deploy" && { + ceph-deploy new "${hosts[@]}" + echo "public_network = ${net_mask}" >> "$HOME/ceph-deploy/ceph.conf" + ceph-deploy mon create-initial + ceph-deploy --overwrite-conf admin "${hosts[@]}" + ceph-deploy mgr create "${hosts[@]}" + for host in "${hosts[@]}"; do + ceph-deploy osd create --data "$disk_device" "$host" + done + wait + ceph osd pool create ceph-osd-pool 128 +} -- 2.23.0

2 2

[PATCH compass-ci] service/scheduler: print access logs in JSON format
by Wu Zhende 14 Apr '21

14 Apr '21

Kermal log format is as follows: 172.17.0.1 - - [13/Apr/2021:07:53:18 +0800] "GET/~lkp/cgi-bin/lkp-post-run?job_id=crystal.673&source=lifecycle&job_ste=timeout HTTP/1.1" 200 4 "-" "Crystal" change to json: {"level_num":1,"level":"INFO","time":"2021-04-13T16:29:15.064+0800","from":"172.168.131.113:60512","message":"access_record","status_code":200,"method":"PUT","resource":"el_host2queues?host=dc-8g.wuzhende-361524","elapsed":"496.32µs"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 - src/scheduler/scheduler.cr | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..58ab8e6 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -54,7 +54,6 @@ class Sched end def alive(version) - debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" rescue e @log.warn(e) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index ee13fd3..ae30937 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -31,6 +31,7 @@ require "../lib/json_logger" # module Scheduler VERSION = "0.2.0" + logging false add_context_storage_type(Time::Span) @@ -43,6 +44,7 @@ module Scheduler after_all do |env| env.sched.etcd_close + env.log.info(%({"from": "#{env.request.remote_address}", "message": "access_record"})) end # echo alive -- 2.23.0

2 1

[PATCH compass-ci 1/2] container/srv-http: add /srv/initrd/osimage
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, can download osimage to run qemu kvm when os_mount=initramfs, and add env SRV_HTTP_INITRD_PORT Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start-initrd | 1 + sparrow/3-code/dev-env | 2 ++ 2 files changed, 3 insertions(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 93be548..457465b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -14,6 +14,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro + -v /srv/initrd/osimage:/srv/initrd/osimage:ro -v /etc/localtime:/etc/localtime:ro -d srv-http diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 3aeba90..bb576c7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${srv_http_git_host:=$server_ip} : ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} +: ${SRV_HTTP_INITRD_PORT:=20008} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} @@ -92,6 +93,7 @@ SRV_HTTP_OS_HOST: $srv_http_os_host SRV_HTTP_GIT_HOST: $srv_http_git_host SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_INITRD_PORT: $SRV_HTTP_INITRD_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin: add download-osimage script
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, if testbox=vm-*, default os_mount=initramfs, so need to use osimage instead of os rootfs, delete download-rootfs script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-osimage | 36 ++++++++++++++++++++++++++++++++++++ sbin/download-rootfs | 40 ---------------------------------------- sparrow/5-build/os-ready | 2 +- 3 files changed, 37 insertions(+), 41 deletions(-) create mode 100755 sbin/download-osimage delete mode 100755 sbin/download-rootfs diff --git a/sbin/download-osimage b/sbin/download-osimage new file mode 100755 index 0000000..f041846 --- /dev/null +++ b/sbin/download-osimage @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download osimage. +# 1. download default osimage openeuler aarch64 20.03 +# example usage: +# ./download-osimage +# 2. download osimage you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-osimage debian aarch64 sid + +os=${1:-openeuler} +os_arch=${2:-$(arch)} +os_version=${3:-20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/initrd/osimage/${os}/${os_arch}/${os_version} + +download_osimage() +{ + mkdir -p /srv/$rootfs_dir + cd $rootfs_dir + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/cur… + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/run… +} + +is_virtual() +{ + dmidecode -s system-product-name | grep -iq "virtual" +} + +is_virtual || download_osimage diff --git a/sbin/download-rootfs b/sbin/download-rootfs deleted file mode 100755 index f8c245d..0000000 --- a/sbin/download-rootfs +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -# download os rootfs. -# 1. download default os rootfs openeuler aarch64 20.03 -# example usage: -# ./download-rootfs -# 2. download os rootfs you want by add params $os, $os_arch, $os_version. -# example usage: -# ./download-rootfs debian aarch64 sid - -os=${1:-openeuler} -os_arch=${2:-$(arch)} -os_version=${3:-20.03} - -. ${CCI_SRC}/container/defconfig.sh -. ${CCI_SRC}/lib/log.sh - -load_cci_defaults - -rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} - -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir - - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url - - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } - - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz -} - -download_os_rootfs diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 5e1acc4..d4ed2b8 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -4,4 +4,4 @@ . $CCI_SRC/lib/check_public_network.sh -public_network_ok && $CCI_SRC/sbin/download-rootfs +public_network_ok && $CCI_SRC/sbin/download-osimage -- 2.23.0

2 2

[PATCH compass-ci] sbin: add config account yaml script
by Cao Xueliang 14 Apr '21

14 Apr '21

delimiter/delimiter.cr needs it to create delimiter account. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/config_account_yaml.rb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 sbin/config_account_yaml.rb diff --git a/sbin/config_account_yaml.rb b/sbin/config_account_yaml.rb new file mode 100755 index 0000000..a555276 --- /dev/null +++ b/sbin/config_account_yaml.rb @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "../lib/config_account" + +config_yaml ARGV[0] -- 2.23.0

1 0

[PATCH compass-ci] container/etcd: use a scheduled task for compacting in etcd
by Cao Xueliang 14 Apr '21

14 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 6 ++++++ container/etcd/entry.sh | 8 ++++++++ container/etcd/etcd_compact.sh | 7 +++++++ container/etcd/start | 2 +- 4 files changed, 22 insertions(+), 1 deletion(-) create mode 100755 container/etcd/entry.sh create mode 100755 container/etcd/etcd_compact.sh diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index b3957e8..f4746ab 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -14,3 +14,9 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* VOLUME /data + +COPY etcd_compact.sh / +COPY entry.sh / +RUN echo '* 0 * * * /test.sh > /etcd_compact.log 2>&1' >> /var/spool/cron/crontabs/root + +ENTRYPOINT ["/entry.sh"] diff --git a/container/etcd/entry.sh b/container/etcd/entry.sh new file mode 100755 index 0000000..4bfea1e --- /dev/null +++ b/container/etcd/entry.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +crond +/bin/etcd +exec "$@" + diff --git a/container/etcd/etcd_compact.sh b/container/etcd/etcd_compact.sh new file mode 100755 index 0000000..acc20d0 --- /dev/null +++ b/container/etcd/etcd_compact.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +rev=$(etcdctl endpoint status --write-out="json"|egrep -o '"revision":[0-9]*'|egrep -o '[0-9].*') +etcdctl compact $rev && etcdctl defrag + diff --git a/container/etcd/start b/container/etcd/start index db94833..6d32cd2 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -14,10 +14,10 @@ cmd=( -p 4001:4001 -p 7001:7001 -v /srv/etcd/dir:/data + -v /etc/localtime:/etc/localtime:ro -e ETCD_UNSUPPORTED_ARCH=arm64 --name etcd-server etcd-v3.4.15:alpine - /bin/etcd --name node-1 --data-dir /data --initial-cluster-state new -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: set default values
by Liu Yinsi 14 Apr '21

14 Apr '21

before: submit host-info.yaml os_arch=x86_64 arch=x86_64 after : submit host-info.yaml [why] when submit job in x86_64 machine, must execute like this: `submit host-info.yaml os_arch=x86_64 arch=x86_64`, then the job can be consumed by docker testbox, so add default values, it will be more convenient. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index d2d9bec76..47b377001 100755 --- a/sbin/submit +++ b/sbin/submit @@ -10,7 +10,10 @@ require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' -opt_set_key_value = {} +opt_set_key_value = { + 'os_arch' => `arch`.chomp, + 'arch' => `arch`.chomp +} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false -- 2.23.0

3 4