- Compass-ci - mailweb.openeuler.org

review star
by Wu Fengguang 15 Apr '21

15 Apr '21

: wqUser-Agent: Mutt/1.10.1 (2018-07-13) User-Agent: Mutt/1.10.1 (2018-07-13) Cao Xueliang is our review star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 75 4 4 15 Cao Xueliang 15 4 10 12 Yu Chuan 21 5 5 8 Wei Jihui 24 6 11 6 Liu Shaofei 11 7 16 4 Luan Shengde 3 1 3 3 Xu Xijian 13 3 3 1 Lin Jiaxin 4 3 6 1 Li Yuanchao 12 1 1 1 Bai Jing 9 0 0 0 Wu Zhende 14 0 0 0 Cui Yili 9 1 8 0 Wang Yong 3 0 0 0 Li Ping 10 2 3 0 Lu Weitao 5 0 0 0 Wu Fengguang 21 0 0 0 Liu Yinsi 11 0 0 0 Wang Chenglong

1 0

git count
by Wu Fengguang 15 Apr '21

15 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 13 Cao Xueliang 7 Wu Zhende 6 Liu Shaofei 6 Lu Weitao 6 Liu Yinsi 4 Wei Jihui 3 Bai Jing 2 Xu Xijian 2 Cui Yili 2 Wang Yong 2 Li Yuanchao 2 Lin Jiaxin 1 Yu Chuan 1 Wang Chenglong 1 Luan Shengde 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan 0 Li Ping

1 0

[PATCH lkp-tests] setup cluster for multi server(3 or more)
by Luan Shengde 15 Apr '21

15 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- cluster/cs-s1-a122-c2 | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 cluster/cs-s1-a122-c2 diff --git a/cluster/cs-s1-a122-c2 b/cluster/cs-s1-a122-c2 new file mode 100644 index 000000000..a69179eab --- /dev/null +++ b/cluster/cs-s1-a122-c2 @@ -0,0 +1,14 @@ +switch: Switch-P12 +ip0: 1 +nodes: + taishan200-2280-2s64p-256g--a122: + roles: [ node1 ] + macs: [ "44:a1:91:9d:d8:25" ] + + taishan200-2280-2s64p-256g--a123: + roles: [ node2 ] + macs: [ "44:a1:91:9d:f5:4a" ] + + taishan200-2280-2s64p-256g--a125: + roles: [ node3 ] + macs: [ "44:67:47:85:f4:65" ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: matching url is not printed in function replace_source
by Lin Jiaxin 15 Apr '21

15 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 350d8fdf..b399fdc3 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -122,7 +122,7 @@ replace_source() echo "source=(" > $1 for url in ${source[@]} do - echo "$url" | egrep 'git\+|\.git$' && { + echo "$url" | egrep 'git\+|\.git$' > /dev/null && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 15 Apr '21

15 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg.yaml" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

2 2

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 14 Apr '21

14 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

1 0

[PATCH lkp-tests] tests: script to deploy ceph for multi server test
by Luan Shengde 14 Apr '21

14 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/ceph-init | 152 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100755 tests/ceph-init diff --git a/tests/ceph-init b/tests/ceph-init new file mode 100755 index 000000000..735376df6 --- /dev/null +++ b/tests/ceph-init @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# - cephuser_password +# - cephuser + +set -x + +: "${cephuser:=cephuser}" +: "${cephuser_password:=cephuser_password}" + +### ===== on each node ===== +# stop firewalld and disable selinx +systemctl stop firewalld +systemctl disable firewalld + +sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux + +until [[ -n $node_roles ]]; do + sleep 1 +done + +hosts=(cephnode1 cephnode2 cephnode3) + +[[ $node_roles == cephnode1 ]] && { + hostnamectl set-hostname cephnode1 +} + +[[ $node_roles == cephnode2 ]] && { + hostnamectl set-hostname cephnode2 +} + +[[ $node_roles == cephnode3 ]] && { + hostnamectl set-hostname cephnode3 +} + +yum update +yum install ntp wget gcc make ceph lvm2 python3 -y + +# add user and set sudo permissions +useradd -d /home/$cephuser -m $cephuser +echo $cephuser:$cephuser_password | chpasswd + +cat >> /etc/sudoers <<-EOF +$cephuser ALL=(ALL) NOPASSWD:ALL +EOF + +# check and install pip +[[ $(command -v pip) ]] || { + until wget https://bootstrap.pypa.io/get-pip.py; do + sleep 2 + done + python get-pip.py +} + +pip install prettytable + +### ===== on ceph-deploy node ===== +# get node IPs is first +[[ $node_roles == cephnode1 ]] || exit 0 + +until [[ -n $cephnode1 ]]; do + sleep 1 +done + +until [[ -n $cephnode2 ]]; do + sleep 1 +done + +until [[ -n $cephnode3 ]]; do + sleep 1 +done + +# config hosts and set default ssh users +cat >> /etc/hosts <<-EOF +$cephnode1 cephnode1 cephnode1 +$cephnode2 cephnode2 cephnode2 +$cephnode3 cephnode3 cephnode3 +EOF + +for host in "${hosts[@]}"; do + cat >> "$HOME/.ssh/config" <<-EOF + Host $host + Hostname $host + User $cephuser + EOF +done + +chmod 0600 "$HOME/.ssh/config" + + +network=$(ip route | grep default | awk '{print $3}' | awk -F'.' '{print $1"."$2}') +net_mask=$(ip route | grep "^$network" | awk '{print $1}') + +# do the deploy until all the servers are up +# for host in "${host[@]}"; do +# until ping -c 1 -W 1 172.168.131.100> /dev/null; do +# sleep 2 +# done +# done + +# set server trust +[[ $(command -v sshpass) ]] || { + until wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz; do + sleep 2 + done +} + +tar xzvf sshpass.tar.gz && { + cd sshpass-1.08 || exit 1 + ./configure + make + make install +} + +rm -rf "$HOME/.ssh/id_rsa" "$HOME/.ssh/id_rsa.pub" +ssh-keygen -f "$HOME/.ssh/id_rsa" -t rsa -N '' + +sed -i '/StrictHostKeyChecking/c StrictHostKeyChecking no' /etc/ssh/ssh_config +for host in "${hosts[@]}"; do +{ + sshpass -p $cephuser_password ssh-copy-id "$cephuser"@"$host" &> /dev/null +}& +done +wait + +# install +pip install ceph-deploy + +pip_dir=$(pip show ceph-deploy | awk '/^Location/ {print $NF}') +ceph_init_file="${pip_dir}/ceph_deploy/hosts/__init__.py" +[[ -f $ceph_init_file ]]&& { + sed -i "/distributions = {/a 'openeuler': fedora," "$ceph_init_file" +} + +# deploy ceph cluster +disk_name=$(sed -e '1d' -e '/^$/d' /proc/partitions | grep " .da" | awk '{print $NF}') +disk_device="/dev/${disk_name}" + +mkdir -p "$HOME/ceph-deploy" +cd "$HOME/ceph-deploy" && { + ceph-deploy new "${hosts[@]}" + echo "public_network = ${net_mask}" >> "$HOME/ceph-deploy/ceph.conf" + ceph-deploy mon create-initial + ceph-deploy --overwrite-conf admin "${hosts[@]}" + ceph-deploy mgr create "${hosts[@]}" + for host in "${hosts[@]}"; do + ceph-deploy osd create --data "$disk_device" "$host" + done + wait + ceph osd pool create ceph-osd-pool 128 +} -- 2.23.0

2 2

[PATCH compass-ci] service/scheduler: print access logs in JSON format
by Wu Zhende 14 Apr '21

14 Apr '21

Kermal log format is as follows: 172.17.0.1 - - [13/Apr/2021:07:53:18 +0800] "GET/~lkp/cgi-bin/lkp-post-run?job_id=crystal.673&source=lifecycle&job_ste=timeout HTTP/1.1" 200 4 "-" "Crystal" change to json: {"level_num":1,"level":"INFO","time":"2021-04-13T16:29:15.064+0800","from":"172.168.131.113:60512","message":"access_record","status_code":200,"method":"PUT","resource":"el_host2queues?host=dc-8g.wuzhende-361524","elapsed":"496.32µs"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 - src/scheduler/scheduler.cr | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..58ab8e6 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -54,7 +54,6 @@ class Sched end def alive(version) - debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" rescue e @log.warn(e) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index ee13fd3..ae30937 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -31,6 +31,7 @@ require "../lib/json_logger" # module Scheduler VERSION = "0.2.0" + logging false add_context_storage_type(Time::Span) @@ -43,6 +44,7 @@ module Scheduler after_all do |env| env.sched.etcd_close + env.log.info(%({"from": "#{env.request.remote_address}", "message": "access_record"})) end # echo alive -- 2.23.0

2 1

[PATCH compass-ci 1/2] container/srv-http: add /srv/initrd/osimage
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, can download osimage to run qemu kvm when os_mount=initramfs, and add env SRV_HTTP_INITRD_PORT Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start-initrd | 1 + sparrow/3-code/dev-env | 2 ++ 2 files changed, 3 insertions(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 93be548..457465b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -14,6 +14,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro + -v /srv/initrd/osimage:/srv/initrd/osimage:ro -v /etc/localtime:/etc/localtime:ro -d srv-http diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 3aeba90..bb576c7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${srv_http_git_host:=$server_ip} : ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} +: ${SRV_HTTP_INITRD_PORT:=20008} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} @@ -92,6 +93,7 @@ SRV_HTTP_OS_HOST: $srv_http_os_host SRV_HTTP_GIT_HOST: $srv_http_git_host SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_INITRD_PORT: $SRV_HTTP_INITRD_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin: add download-osimage script
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, if testbox=vm-*, default os_mount=initramfs, so need to use osimage instead of os rootfs, delete download-rootfs script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-osimage | 36 ++++++++++++++++++++++++++++++++++++ sbin/download-rootfs | 40 ---------------------------------------- sparrow/5-build/os-ready | 2 +- 3 files changed, 37 insertions(+), 41 deletions(-) create mode 100755 sbin/download-osimage delete mode 100755 sbin/download-rootfs diff --git a/sbin/download-osimage b/sbin/download-osimage new file mode 100755 index 0000000..f041846 --- /dev/null +++ b/sbin/download-osimage @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download osimage. +# 1. download default osimage openeuler aarch64 20.03 +# example usage: +# ./download-osimage +# 2. download osimage you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-osimage debian aarch64 sid + +os=${1:-openeuler} +os_arch=${2:-$(arch)} +os_version=${3:-20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/initrd/osimage/${os}/${os_arch}/${os_version} + +download_osimage() +{ + mkdir -p /srv/$rootfs_dir + cd $rootfs_dir + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/cur… + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/run… +} + +is_virtual() +{ + dmidecode -s system-product-name | grep -iq "virtual" +} + +is_virtual || download_osimage diff --git a/sbin/download-rootfs b/sbin/download-rootfs deleted file mode 100755 index f8c245d..0000000 --- a/sbin/download-rootfs +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -# download os rootfs. -# 1. download default os rootfs openeuler aarch64 20.03 -# example usage: -# ./download-rootfs -# 2. download os rootfs you want by add params $os, $os_arch, $os_version. -# example usage: -# ./download-rootfs debian aarch64 sid - -os=${1:-openeuler} -os_arch=${2:-$(arch)} -os_version=${3:-20.03} - -. ${CCI_SRC}/container/defconfig.sh -. ${CCI_SRC}/lib/log.sh - -load_cci_defaults - -rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} - -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir - - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url - - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } - - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz -} - -download_os_rootfs diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 5e1acc4..d4ed2b8 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -4,4 +4,4 @@ . $CCI_SRC/lib/check_public_network.sh -public_network_ok && $CCI_SRC/sbin/download-rootfs +public_network_ok && $CCI_SRC/sbin/download-osimage -- 2.23.0

2 2