- Compass-ci - mailweb.openeuler.org

review star
by Wu Fengguang 15 Apr '21

15 Apr '21

: wqUser-Agent: Mutt/1.10.1 (2018-07-13) User-Agent: Mutt/1.10.1 (2018-07-13) Cao Xueliang is our review star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 75 4 4 15 Cao Xueliang 15 4 10 12 Yu Chuan 21 5 5 8 Wei Jihui 24 6 11 6 Liu Shaofei 11 7 16 4 Luan Shengde 3 1 3 3 Xu Xijian 13 3 3 1 Lin Jiaxin 4 3 6 1 Li Yuanchao 12 1 1 1 Bai Jing 9 0 0 0 Wu Zhende 14 0 0 0 Cui Yili 9 1 8 0 Wang Yong 3 0 0 0 Li Ping 10 2 3 0 Lu Weitao 5 0 0 0 Wu Fengguang 21 0 0 0 Liu Yinsi 11 0 0 0 Wang Chenglong

1 0

git count
by Wu Fengguang 15 Apr '21

15 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 13 Cao Xueliang 7 Wu Zhende 6 Liu Shaofei 6 Lu Weitao 6 Liu Yinsi 4 Wei Jihui 3 Bai Jing 2 Xu Xijian 2 Cui Yili 2 Wang Yong 2 Li Yuanchao 2 Lin Jiaxin 1 Yu Chuan 1 Wang Chenglong 1 Luan Shengde 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan 0 Li Ping

1 0

[PATCH lkp-tests] setup cluster for multi server(3 or more)
by Luan Shengde 15 Apr '21

15 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- cluster/cs-s1-a122-c2 | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 cluster/cs-s1-a122-c2 diff --git a/cluster/cs-s1-a122-c2 b/cluster/cs-s1-a122-c2 new file mode 100644 index 000000000..a69179eab --- /dev/null +++ b/cluster/cs-s1-a122-c2 @@ -0,0 +1,14 @@ +switch: Switch-P12 +ip0: 1 +nodes: + taishan200-2280-2s64p-256g--a122: + roles: [ node1 ] + macs: [ "44:a1:91:9d:d8:25" ] + + taishan200-2280-2s64p-256g--a123: + roles: [ node2 ] + macs: [ "44:a1:91:9d:f5:4a" ] + + taishan200-2280-2s64p-256g--a125: + roles: [ node3 ] + macs: [ "44:67:47:85:f4:65" ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: matching url is not printed in function replace_source
by Lin Jiaxin 15 Apr '21

15 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 350d8fdf..b399fdc3 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -122,7 +122,7 @@ replace_source() echo "source=(" > $1 for url in ${source[@]} do - echo "$url" | egrep 'git\+|\.git$' && { + echo "$url" | egrep 'git\+|\.git$' > /dev/null && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 15 Apr '21

15 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg.yaml" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

2 2

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: remove upstream_repo
by Lin Jiaxin 14 Apr '21

14 Apr '21

Assigning a null value to upstream_repo is meaningless. And when use "submit upstream_repo=xxx rpmbuild-pkg" will make upstream_repo be null. Unless we put "upstream_repo=xxx" behind. Or declare it in yaml. [example] linjiaxin@crystal ~%/home/linjiaxin/lkp-tests/sbin/submit upstream_repo="a/azk/azk" upstream_url=https://github.com/azukiapp/azk pkgbuild_repo=archlinux/aur-a/azk-git testbox=dc-16g os=openeuler os_version=20.03-LTS os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-LTS repo_name=aur rpmbuild-pkg.yaml upstream_branch=master upstream_commit=e72adb039e4dd64fd6ed81d82fd597c1d4efdf14 upstream_dir=upstream submit_id=d18fabcd-6302-464c-aa99-3ed9e610704b submit /home/linjiaxin/lkp-tests/jobs/rpmbuild-pkg.yaml, got job id=crystal.1594407 linjiaxin@crystal ~% es-find id=crystal.1594407 | grep upstream_repo "upstream_repo": null, Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index a70590af..5860ba02 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -upstream_repo: rpmbuild-pkg: -- 2.23.0

1 0

[PATCH lkp-tests] tests: script to deploy ceph for multi server test
by Luan Shengde 14 Apr '21

14 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/ceph-init | 152 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100755 tests/ceph-init diff --git a/tests/ceph-init b/tests/ceph-init new file mode 100755 index 000000000..735376df6 --- /dev/null +++ b/tests/ceph-init @@ -0,0 +1,152 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# - cephuser_password +# - cephuser + +set -x + +: "${cephuser:=cephuser}" +: "${cephuser_password:=cephuser_password}" + +### ===== on each node ===== +# stop firewalld and disable selinx +systemctl stop firewalld +systemctl disable firewalld + +sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux + +until [[ -n $node_roles ]]; do + sleep 1 +done + +hosts=(cephnode1 cephnode2 cephnode3) + +[[ $node_roles == cephnode1 ]] && { + hostnamectl set-hostname cephnode1 +} + +[[ $node_roles == cephnode2 ]] && { + hostnamectl set-hostname cephnode2 +} + +[[ $node_roles == cephnode3 ]] && { + hostnamectl set-hostname cephnode3 +} + +yum update +yum install ntp wget gcc make ceph lvm2 python3 -y + +# add user and set sudo permissions +useradd -d /home/$cephuser -m $cephuser +echo $cephuser:$cephuser_password | chpasswd + +cat >> /etc/sudoers <<-EOF +$cephuser ALL=(ALL) NOPASSWD:ALL +EOF + +# check and install pip +[[ $(command -v pip) ]] || { + until wget https://bootstrap.pypa.io/get-pip.py; do + sleep 2 + done + python get-pip.py +} + +pip install prettytable + +### ===== on ceph-deploy node ===== +# get node IPs is first +[[ $node_roles == cephnode1 ]] || exit 0 + +until [[ -n $cephnode1 ]]; do + sleep 1 +done + +until [[ -n $cephnode2 ]]; do + sleep 1 +done + +until [[ -n $cephnode3 ]]; do + sleep 1 +done + +# config hosts and set default ssh users +cat >> /etc/hosts <<-EOF +$cephnode1 cephnode1 cephnode1 +$cephnode2 cephnode2 cephnode2 +$cephnode3 cephnode3 cephnode3 +EOF + +for host in "${hosts[@]}"; do + cat >> "$HOME/.ssh/config" <<-EOF + Host $host + Hostname $host + User $cephuser + EOF +done + +chmod 0600 "$HOME/.ssh/config" + + +network=$(ip route | grep default | awk '{print $3}' | awk -F'.' '{print $1"."$2}') +net_mask=$(ip route | grep "^$network" | awk '{print $1}') + +# do the deploy until all the servers are up +# for host in "${host[@]}"; do +# until ping -c 1 -W 1 172.168.131.100> /dev/null; do +# sleep 2 +# done +# done + +# set server trust +[[ $(command -v sshpass) ]] || { + until wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz; do + sleep 2 + done +} + +tar xzvf sshpass.tar.gz && { + cd sshpass-1.08 || exit 1 + ./configure + make + make install +} + +rm -rf "$HOME/.ssh/id_rsa" "$HOME/.ssh/id_rsa.pub" +ssh-keygen -f "$HOME/.ssh/id_rsa" -t rsa -N '' + +sed -i '/StrictHostKeyChecking/c StrictHostKeyChecking no' /etc/ssh/ssh_config +for host in "${hosts[@]}"; do +{ + sshpass -p $cephuser_password ssh-copy-id "$cephuser"@"$host" &> /dev/null +}& +done +wait + +# install +pip install ceph-deploy + +pip_dir=$(pip show ceph-deploy | awk '/^Location/ {print $NF}') +ceph_init_file="${pip_dir}/ceph_deploy/hosts/__init__.py" +[[ -f $ceph_init_file ]]&& { + sed -i "/distributions = {/a 'openeuler': fedora," "$ceph_init_file" +} + +# deploy ceph cluster +disk_name=$(sed -e '1d' -e '/^$/d' /proc/partitions | grep " .da" | awk '{print $NF}') +disk_device="/dev/${disk_name}" + +mkdir -p "$HOME/ceph-deploy" +cd "$HOME/ceph-deploy" && { + ceph-deploy new "${hosts[@]}" + echo "public_network = ${net_mask}" >> "$HOME/ceph-deploy/ceph.conf" + ceph-deploy mon create-initial + ceph-deploy --overwrite-conf admin "${hosts[@]}" + ceph-deploy mgr create "${hosts[@]}" + for host in "${hosts[@]}"; do + ceph-deploy osd create --data "$disk_device" "$host" + done + wait + ceph osd pool create ceph-osd-pool 128 +} -- 2.23.0

2 2

[PATCH compass-ci] service/scheduler: print access logs in JSON format
by Wu Zhende 14 Apr '21

14 Apr '21

Kermal log format is as follows: 172.17.0.1 - - [13/Apr/2021:07:53:18 +0800] "GET/~lkp/cgi-bin/lkp-post-run?job_id=crystal.673&source=lifecycle&job_ste=timeout HTTP/1.1" 200 4 "-" "Crystal" change to json: {"level_num":1,"level":"INFO","time":"2021-04-13T16:29:15.064+0800","from":"172.168.131.113:60512","message":"access_record","status_code":200,"method":"PUT","resource":"el_host2queues?host=dc-8g.wuzhende-361524","elapsed":"496.32µs"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 - src/scheduler/scheduler.cr | 2 ++ 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..58ab8e6 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -54,7 +54,6 @@ class Sched end def alive(version) - debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" rescue e @log.warn(e) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index ee13fd3..ae30937 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -31,6 +31,7 @@ require "../lib/json_logger" # module Scheduler VERSION = "0.2.0" + logging false add_context_storage_type(Time::Span) @@ -43,6 +44,7 @@ module Scheduler after_all do |env| env.sched.etcd_close + env.log.info(%({"from": "#{env.request.remote_address}", "message": "access_record"})) end # echo alive -- 2.23.0

2 1

[PATCH compass-ci 1/2] container/srv-http: add /srv/initrd/osimage
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, can download osimage to run qemu kvm when os_mount=initramfs, and add env SRV_HTTP_INITRD_PORT Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start-initrd | 1 + sparrow/3-code/dev-env | 2 ++ 2 files changed, 3 insertions(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 93be548..457465b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -14,6 +14,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro + -v /srv/initrd/osimage:/srv/initrd/osimage:ro -v /etc/localtime:/etc/localtime:ro -d srv-http diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 3aeba90..bb576c7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${srv_http_git_host:=$server_ip} : ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} +: ${SRV_HTTP_INITRD_PORT:=20008} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} @@ -92,6 +93,7 @@ SRV_HTTP_OS_HOST: $srv_http_os_host SRV_HTTP_GIT_HOST: $srv_http_git_host SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_INITRD_PORT: $SRV_HTTP_INITRD_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin: add download-osimage script
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, if testbox=vm-*, default os_mount=initramfs, so need to use osimage instead of os rootfs, delete download-rootfs script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-osimage | 36 ++++++++++++++++++++++++++++++++++++ sbin/download-rootfs | 40 ---------------------------------------- sparrow/5-build/os-ready | 2 +- 3 files changed, 37 insertions(+), 41 deletions(-) create mode 100755 sbin/download-osimage delete mode 100755 sbin/download-rootfs diff --git a/sbin/download-osimage b/sbin/download-osimage new file mode 100755 index 0000000..f041846 --- /dev/null +++ b/sbin/download-osimage @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download osimage. +# 1. download default osimage openeuler aarch64 20.03 +# example usage: +# ./download-osimage +# 2. download osimage you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-osimage debian aarch64 sid + +os=${1:-openeuler} +os_arch=${2:-$(arch)} +os_version=${3:-20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/initrd/osimage/${os}/${os_arch}/${os_version} + +download_osimage() +{ + mkdir -p /srv/$rootfs_dir + cd $rootfs_dir + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/cur… + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/run… +} + +is_virtual() +{ + dmidecode -s system-product-name | grep -iq "virtual" +} + +is_virtual || download_osimage diff --git a/sbin/download-rootfs b/sbin/download-rootfs deleted file mode 100755 index f8c245d..0000000 --- a/sbin/download-rootfs +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -# download os rootfs. -# 1. download default os rootfs openeuler aarch64 20.03 -# example usage: -# ./download-rootfs -# 2. download os rootfs you want by add params $os, $os_arch, $os_version. -# example usage: -# ./download-rootfs debian aarch64 sid - -os=${1:-openeuler} -os_arch=${2:-$(arch)} -os_version=${3:-20.03} - -. ${CCI_SRC}/container/defconfig.sh -. ${CCI_SRC}/lib/log.sh - -load_cci_defaults - -rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} - -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir - - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url - - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } - - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz -} - -download_os_rootfs diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 5e1acc4..d4ed2b8 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -4,4 +4,4 @@ . $CCI_SRC/lib/check_public_network.sh -public_network_ok && $CCI_SRC/sbin/download-rootfs +public_network_ok && $CCI_SRC/sbin/download-osimage -- 2.23.0

2 2

[PATCH compass-ci] sbin: add config account yaml script
by Cao Xueliang 14 Apr '21

14 Apr '21

delimiter/delimiter.cr needs it to create delimiter account. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/config_account_yaml.rb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 sbin/config_account_yaml.rb diff --git a/sbin/config_account_yaml.rb b/sbin/config_account_yaml.rb new file mode 100755 index 0000000..a555276 --- /dev/null +++ b/sbin/config_account_yaml.rb @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "../lib/config_account" + +config_yaml ARGV[0] -- 2.23.0

1 0

[PATCH compass-ci] container/etcd: use a scheduled task for compacting in etcd
by Cao Xueliang 14 Apr '21

14 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 6 ++++++ container/etcd/entry.sh | 8 ++++++++ container/etcd/etcd_compact.sh | 7 +++++++ container/etcd/start | 2 +- 4 files changed, 22 insertions(+), 1 deletion(-) create mode 100755 container/etcd/entry.sh create mode 100755 container/etcd/etcd_compact.sh diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index b3957e8..f4746ab 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -14,3 +14,9 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* VOLUME /data + +COPY etcd_compact.sh / +COPY entry.sh / +RUN echo '* 0 * * * /test.sh > /etcd_compact.log 2>&1' >> /var/spool/cron/crontabs/root + +ENTRYPOINT ["/entry.sh"] diff --git a/container/etcd/entry.sh b/container/etcd/entry.sh new file mode 100755 index 0000000..4bfea1e --- /dev/null +++ b/container/etcd/entry.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +crond +/bin/etcd +exec "$@" + diff --git a/container/etcd/etcd_compact.sh b/container/etcd/etcd_compact.sh new file mode 100755 index 0000000..acc20d0 --- /dev/null +++ b/container/etcd/etcd_compact.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +rev=$(etcdctl endpoint status --write-out="json"|egrep -o '"revision":[0-9]*'|egrep -o '[0-9].*') +etcdctl compact $rev && etcdctl defrag + diff --git a/container/etcd/start b/container/etcd/start index db94833..6d32cd2 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -14,10 +14,10 @@ cmd=( -p 4001:4001 -p 7001:7001 -v /srv/etcd/dir:/data + -v /etc/localtime:/etc/localtime:ro -e ETCD_UNSUPPORTED_ARCH=arm64 --name etcd-server etcd-v3.4.15:alpine - /bin/etcd --name node-1 --data-dir /data --initial-cluster-state new -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: set default values
by Liu Yinsi 14 Apr '21

14 Apr '21

before: submit host-info.yaml os_arch=x86_64 arch=x86_64 after : submit host-info.yaml [why] when submit job in x86_64 machine, must execute like this: `submit host-info.yaml os_arch=x86_64 arch=x86_64`, then the job can be consumed by docker testbox, so add default values, it will be more convenient. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index d2d9bec76..47b377001 100755 --- a/sbin/submit +++ b/sbin/submit @@ -10,7 +10,10 @@ require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' -opt_set_key_value = {} +opt_set_key_value = { + 'os_arch' => `arch`.chomp, + 'arch' => `arch`.chomp +} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false -- 2.23.0

3 4

[PATCH compass-ci] container/dracut-initrd: set yes for choices when pvcreate and vgcreate
by Xu Xijian 13 Apr '21

13 Apr '21

[why] When create physical volume, some choices need to be made, we should set it as yes(-y) to keep going instead of skipping [errlog] [ 260.644037] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): lvm pvcreate /dev/sda [ 260.664078] dracut-pre-mount[4008]: WARNING: ext4 signature detected on /dev/sda at offset 1080. Wipe it? [y/n]: [n] [ 260.684036] dracut-pre-mount[4008]: Aborted wiping of ext4. [ 260.700034] dracut-pre-mount[4008]: 1 existing signature left on the device. [ 260.716076] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): reboot Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 2a07270..9e83742 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -29,13 +29,13 @@ sync_src_lv() { # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot } } -- 2.23.0

1 0

[PATCH compass-ci] es-jobs: reuse results of other programs
by Wu Zhende 13 Apr '21

13 Apr '21

[Why] es-jobs's result may need use in other programs. But it's hard to get it right now. [How] use: es_jobs.generate_result can get the hash result Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 6 +++++- sbin/es-jobs | 1 + 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index b0d0fb2..25e54b4 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -168,6 +168,10 @@ class ESJobs end def output + output_yaml('', @result) + end + + def generate_result if @jobs.empty? puts "No query result is found: #{@es_query}" return @@ -175,6 +179,6 @@ class ESJobs @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h - output_yaml('', @result) + @result end end diff --git a/sbin/es-jobs b/sbin/es-jobs index ff0d540..6d17d6b 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -52,4 +52,5 @@ items = parse_argv raise 'Please enter a query' if items.empty? es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.generate_result es_jobs.output -- 2.23.0

1 0

[PATCH compass-ci] lib/sched.cr: change testbox's queues
by Wu Zhende 13 Apr '21

13 Apr '21

[Before] ["sched/dc-8g.aarch64/ready"] [After] ["dc-8g.aarch64"] [Why] "sched/.*/ready" is a fixed format. The length of each column on kibana is limited, and removing the fixed format can also highlinght the point. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..35315ce 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -158,6 +158,16 @@ class Sched spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end + # input: ["sched/dc-8g.aarch64/ready"] + # output: ["dc-8g.aarch64"] + def fetch_queues(queues) + new_queues = [] of String + queues.each do |queue| + new_queues << queue.split("/")[1] + end + new_queues + end + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline @@ -172,7 +182,7 @@ class Sched end type = get_type(testbox) - queues = JSON.parse(queues.to_json) + queues = JSON.parse(fetch_queues(queues).to_json) hash = { "job_id" => job_id, "state" => state, -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 13 Apr '21

13 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

2 4

[PATCH compass-ci] lib/git_mirror.rb: retry es query
by Li Yuanchao 13 Apr '21

13 Apr '21

[errmsg] git-mirror: /usr/lib/ruby/2.5.0/socket.rb:452:in `__read_nonblock': Connection reset by peer (Faraday::ConnectionFailed) Sometimes the error above occurs, and container git-mirror restart. It's not a stable error, and next time es query can be success. So retry here. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bcdb077..6100d6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -432,7 +432,14 @@ class MirrorMain query = { query: { match: { _id: git_repo } } } return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? - result = @es_client.search(index: 'repo', body: query)['hits'] + begin + result = @es_client.search(index: 'repo', body: query)['hits'] + rescue StandardError + puts $ERROR_INFO + sleep 1 + retry + end + fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: post data to rpm-repo docker
by Wang Yong 13 Apr '21

13 Apr '21

post rpm data to rpm-repo in order to createrepo ref: commit 5dd22c1e010734bc5e610f4abe46baec0f36f5bb Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a91ae4a08..312582d16 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -108,6 +108,32 @@ upload_rpm_pkg() done } +handle_rpm_list() +{ + rpm_list=($(show_rpm_files | xargs basename -a)) + srpm_list=($(show_src_rpm_files | xargs basename -a)) + full_list=(${rpm_list[@]} ${srpm_list[@]}) + j=0 + + for rpm_name in ${full_list[@]} + do + full_list[$j]="\"/srv${rpm_dest}/${rpm_name}\"" + [[ ${rpm_name} =~ "src.rpm" ]] && full_list[$j]="\"/srv${src_rpm_dest}/${rpm_name}\"" + ((j++)) + done + + full_list=$(echo ${full_list[@]} | sed 's; ;,;g') +} + +update_repo_mq() +{ + handle_rpm_list + + curl -H 'Content-Type: Application/json' -XPOST "${LKP_SERVER}:10003/upload" \ + -d "{\"upload_rpms\": ["${full_list}"]}" +} + [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg +update_repo_mq -- 2.23.0

1 0

[PATCH compass-ci] container/scheduler: fix Missing ENV key: NODE_PORT
by Cao Xueliang 13 Apr '21

13 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler/my-start b/container/scheduler/my-start index 8d800f9..e5855af 100755 --- a/container/scheduler/my-start +++ b/container/scheduler/my-start @@ -65,6 +65,7 @@ cmd = %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH compass-ci] scheduler: use NODE_PORT to run kemal
by Cao Xueliang 13 Apr '21

13 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 5 +++-- src/scheduler.cr | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index b13b396..598b263 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -7,6 +7,7 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ + SCHED_HOST SCHED_PORT SCHED_NODES REDIS_HOST @@ -38,7 +39,7 @@ defaults['SCHED_PORT'] ||= '3000' defaults['MASTER_FLUENTD_PORT'] ||= '24224' defaults['SRV_HTTP_CCI_PORT'] ||= '11307' sched_nodes = defaults.delete('SCHED_NODES') -SCHED_PORT = defaults.delete('SCHED_PORT') +SCHED_PORT = defaults['SCHED_PORT'] sched_nodes.each do |node| docker_rm "s001-alpine-#{node['port']}" @@ -69,7 +70,7 @@ sched_nodes.each do |node| -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_PORT=#{node_port} + -e NODE_PORT=#{node_port} -v /srv/tmp/public:/c/cci/scheduler/public -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} diff --git a/src/scheduler.cr b/src/scheduler.cr index 07b38a3..4e4bf24 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -9,7 +9,7 @@ module Scheduler log = JSONLogger.new begin - Kemal.run(SCHED_PORT) + Kemal.run(ENV["NODE_PORT"].to_i32) rescue e log.error(e) end -- 2.23.0

2 2

[PATCH compass-ci 2/2] src/extract-stats/stats_worker.cr: optimize output
by Lu Weitao 13 Apr '21

13 Apr '21

[Why] standardized output log for kibana Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index e371bf3..47c3a59 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -2,6 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "../lib/etcd_client" +require "../lib/json_logger" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" require "../scheduler/constants" @@ -16,6 +17,7 @@ class StatsWorker @es = Elasticsearch::Client.new @etcd = EtcdClient.new @rc = RegressionClient.new + @log = JSONLogger.new end def handle(queue_path, channel) @@ -30,7 +32,7 @@ class StatsWorker @etcd.delete(queue_path) rescue e channel.send(queue_path) - STDERR.puts e.message + @log.error(e.message) # incase of many error message when task-queue, ES does not work sleep(10) ensure @@ -95,16 +97,16 @@ class StatsWorker new_error_ids = check_new_error_ids(error_ids, job_id) unless new_error_ids.empty? sample_error_id = new_error_ids.sample - STDOUT.puts "send a delimiter task: job_id is #{job_id}" + @log.info("send a delimiter task: job_id is #{job_id}") queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" value = {"job_id" => job_id, "error_id" => sample_error_id} @etcd.put(queue, value) msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) - system "echo '#{msg}'" + @log.info(msg) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) - system "echo '#{msg}'" + @log.info(msg) end def check_new_error_ids(error_ids : Array, job_id : String) @@ -113,7 +115,7 @@ class StatsWorker begin is_exists = @rc.check_error_id error_id rescue e - STDERR.puts e.message + @log.error(e.message) next end next if is_exists -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/dump_stat.rb: optimize useless special chars
by Lu Weitao 13 Apr '21

13 Apr '21

[Why] we already use strip, so needn't chomp Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 383ff09..04a7632 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -38,7 +38,7 @@ module DumpStat stat_result.each do |key, value| key = key.resolve_invalid_bytes - key = key.chomp.strip + key = key.strip next if key[0] == '#' next if value.empty? || value == 0 next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: delete useless require
by Wei Jihui 13 Apr '21

13 Apr '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 1 - 1 file changed, 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index d2d9bec7..edb207cc 100755 --- a/sbin/submit +++ b/sbin/submit @@ -5,7 +5,6 @@ LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) require "#{LKP_SRC}/lib/job2sh" -require "#{LKP_SRC}/lib/distro_info" require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: post data to rpm-repo docker
by Wang Yong 13 Apr '21

13 Apr '21

post rpm data to rpm-repo in order to createrepo ref: commit 5dd22c1e010734bc5e610f4abe46baec0f36f5bb Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a91ae4a08..3d63bebae 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -108,6 +108,33 @@ upload_rpm_pkg() done } +update_repo_mq() +{ + rpm_list=($(find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" -exec basename {} \;)) + srpm_list=($(find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" -exec basename {} \;)) + j=0 + k=0 + + for rpm_name in ${rpm_list[@]} + do + rpm_list[$j]="\"/srv${rpm_dest}/${rpm_name}\"" + ((j++)) + done + + for src_rpm_name in ${srpm_list[@]} + do + srpm_list[$k]="\"/srv${src_rpm_dest}/${src_rpm_name}\"" + ((k++)) + done + + full_list=(${rpm_list[@]} ${srpm_list[@]}) + full_list=$(echo ${full_list[@]} | sed 's; ;,;g') + + curl -H 'Content-Type: Application/json' -XPOST "${LKP_SERVER}:10003/upload" \ + -d "{\"upload_rpms\": ["${full_list}"]}" +} + [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg +update_repo_mq -- 2.23.0

2 2

[PATCH compass-ci 2/2] sbin/download-rootfs: not to download if machine is virtual
by Liu Yinsi 12 Apr '21

12 Apr '21

[why] we not support to run qemu in a virtual machine, so download os rootfs is unnecessary. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index f8c245d..68a6803 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -37,4 +37,9 @@ download_os_rootfs() rm -rf ${os_version}.cgz } -download_os_rootfs +is_virtual() +{ + dmidecode -s system-product-name | grep -iq "virtual" +} + +is_virtual || download_os_rootfs -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/download-rootfs: download os rootfs according to arch
by Liu Yinsi 12 Apr '21

12 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index 74afbf6..f8c245d 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -10,7 +10,7 @@ # ./download-rootfs debian aarch64 sid os=${1:-openeuler} -os_arch=${2:-aarch64} +os_arch=${2:-$(arch)} os_version=${3:-20.03} . ${CCI_SRC}/container/defconfig.sh -- 2.23.0

1 0

[PATCH lkp-tests] add script for fix dockerfile file
by Bai Jing 12 Apr '21

12 Apr '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/handle_dockerfile.sh | 51 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 lib/handle_dockerfile.sh diff --git a/lib/handle_dockerfile.sh b/lib/handle_dockerfile.sh new file mode 100644 index 000000000..b71f9865d --- /dev/null +++ b/lib/handle_dockerfile.sh @@ -0,0 +1,51 @@ +#!/bin/bash + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$dockerfile_name" +} + +handle_epel() +{ + sed -i "s/epel-release/bash/g" "$dockerfile_name" +} + +handle_epelrpm() +{ + sed -i "s|https://.*epel-release.*rpm|bash|g" "$dockerfile_name" +} + +handle_rpm_gpg() +{ + sed -i "s|RPM-GPG-KEY-CentOS-7|RPM-GPG-KEY-openEuler|g" "$dockerfile_name" + sed -i "s|RPM-GPG-KEY-centosofficial|RPM-GPG-KEY-openEuler|g" "$dockerfile_name" +} + +add_base_commands() +{ + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$dockerfile_name" +} + +handle_epel_repo() +{ + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$dockerfile_name" +} + +handle_error_exit() +{ + sed '/^RUN/RUN set -e;/g' "$dockerfile_name" +} + +handle_dockerfile() +{ + dockerfile_name=$1 + handle_FROM + handle_epel + handle_epelrpm + handle_rpm_gpg + add_base_commands + handle_epel_repo + handle_error_exit +} -- 2.23.0

2 3

help star
by Wu Fengguang 12 Apr '21

12 Apr '21

Yu Chuan is our help star again, thanks Yu Chuan! baijing@crystal ~/todo/bin% ./help-star last 21.4 Wu Fengguang 13.5 Yu Chuan 12.2 Cao Xueliang 10.6 Lin Jiaxin 9.8 Wei Jihui 9.6 Bai Jing 8.6 Wang Yong 7.3 Luan Shengde 7.2 Wu Zhende 7.2 Xu Xijian 6.8 Zhang Dewan 6.3 Liu Yinsi 5.2 Liu Shaofei 4.0 Du Kaitian 3.0 Li Ping 2.6 Li Yuanchao 2.2 Lu Weitao 1.3 Wang Chenglong 1.0 Ren Wen 0.0 Zhang Yale 0.0 Cui Yili

1 0

[PATCH compass-ci] lib/dump_stat.rb: remove left/right space of key
by Lu Weitao 12 Apr '21

12 Apr '21

[Why] 2021-04-09 17:19:16 +0800 WARN -- whitespace in stats name: Installing 2021-04-09 17:19:16 +0800 WARN -- check .//openeuler_docker Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index de0168e..383ff09 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -38,6 +38,7 @@ module DumpStat stat_result.each do |key, value| key = key.resolve_invalid_bytes + key = key.chomp.strip next if key[0] == '#' next if value.empty? || value == 0 next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ -- 2.23.0

2 2

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Apr '21

12 Apr '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 45b75a3..56dad9e 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -62,6 +62,7 @@ class StatsWorker @es.@client.update( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => {:doc => update_content}, } -- 2.23.0

2 1

[PATCH compass-ci] doc: update invalid URL
by Liu Yinsi 12 Apr '21

12 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/manual/apply-account.md | 2 +- doc/manual/apply-account.zh.md | 2 +- doc/tutorial.md | 43 +--------------------------------- 3 files changed, 3 insertions(+), 44 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 542251d..69e678a 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -33,7 +33,7 @@ Email template: The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to add it to compass-ci testing. ## step 2: receive an email diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 849364f..d177a76 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -32,7 +32,7 @@ 您的仓库需要在平台测试覆盖的范围内，并且注册到： [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). 如果贡献的OSS工程仓库不在upstream-repos中，请参考： - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 将仓库注册到compass-ci测试池中。 ## 2. 接收邮件 diff --git a/doc/tutorial.md b/doc/tutorial.md index b04bf70..cd72d5c 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -280,48 +280,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - - >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - +### [本地搭建 compass-ci 服务器节点](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/README… ## 呼吁合作 - 增强 git bisect 能力 -- 2.23.0

1 0

[PATCH lkp-tests] tests/openeuler_docker: optimize file name error
by Liu Shaofei 12 Apr '21

12 Apr '21

[why] ./git--github.com-peternav-myrepo.git/z9.2062565/dockerfile:from_openeuler_1 => ./git--github.com-peternav-myrepo.git/z9.2062565/peternav-myrepo_1 Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/openeuler_docker | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/openeuler_docker b/tests/openeuler_docker index 3ea57e2c9..906d6c88c 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -19,8 +19,10 @@ declare -i dockerfile_count=0 fit_repo() { yum reinstall -y git - git clone "$dockerfile_url" "$BENCHMARK_ROOT/src/" - dockerfile_path="$(find "$BENCHMARK_ROOT/src/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" + project_name="${dockerfile_url##*/}" + project_name="${project_name%.git}" + git clone "$dockerfile_url" "$BENCHMARK_ROOT/{project_name}/" + dockerfile_path="$(find "$BENCHMARK_ROOT/{project_name}/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" dockerfile_count=$(( $(echo $dockerfile_path | awk '{print NF}') / 2 )) [ -n "$dockerfile_path" ] || { echo "current-repo-not-support-centos: 1" @@ -45,7 +47,7 @@ build_centos() build_openeuler() { - docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1 + docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$index 2>&1 [ $? != 0 ] && { let nr_openeuler_fail++ echo "${image}_$index.build.fail: 1" @@ -60,7 +62,7 @@ docker_run() { docker_run_cmd="docker run -d --name ${image}_$1_$index ${image}_$1_$index" [ $1 == "openeuler" ] && { - docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1" + docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$index 2>&1" } eval $docker_run_cmd @@ -84,6 +86,9 @@ handle_build_result() elif [ "$nr_openeuler_success" == "$dockerfile_count" ]; then echo "openeuler_valid_url: $dockerfile_url" exit 0 + else + echo "Build CentOS images failed." + exit 0 fi } @@ -91,7 +96,7 @@ build() { dockerfile_name="${file##*/}" dockerfile_name="${dockerfile_name%:*}" - image=$(echo "$file" | awk -F '/' '{print $5}' | tr 'A-Z' 'a-z') + image=$(echo "$file" | awk -F '/' '$5~/Dockerfile/ {print $4} $5!~/Dockerfile/ {print $5}' | tr 'A-Z' 'a-z') build_centos "centos" > /dev/null 2>&1 || return 1 handle_dockerfile "$dockerfile_name" -- 2.23.0

2 2

git count
by Wu Fengguang 12 Apr '21

12 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 14 Cao Xueliang 8 Wu Zhende 6 Lu Weitao 5 Li Ping 3 Liu Shaofei 3 Li Yuanchao 2 Bai Jing 2 Wei Jihui 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Xu Xijian 1 Wang Chenglong 1 Luan Shengde 1 Zhang Dewan 0 Du Kaitian 0 Liu Yinsi 0 Wu Fengguang 0 Zhang Yale 0 Yu Chuan

1 0

[PATCH compass-ci] sparrow: fix docker pull local repos failed
by Cao Xueliang 12 Apr '21

12 Apr '21

[Error] docker pull 172.168.131.2:5001/openeuler:20.03-LTS Error response from daemon: Get https://172.168.177.252:5001/v2/: http:server gave HTTP response to HTTPS client [why] There is no DOCKER_REGISTRY_HOST and DOCKER_REGISTRY_PORT in mutli-docker job, so use localhost value as the DOCKER_REGISTRY_HOST. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fd529e7..3aeba90 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -149,6 +149,8 @@ ASSIST_RESULT_HOST: ASSIST_RESULT_PORT: RESULT_WEBDAV_HOST: RESULT_WEBDAV_PORT: +DOCKER_REGISTRY_HOST: +DOCKER_REGISTRY_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add check_public_network.sh
by Liu Yinsi 12 Apr '21

12 Apr '21

Check public network ok or not during locally deploy Compass CI, because users need to sparrow in a pure intranet environment. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/check_public_network.sh | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 lib/check_public_network.sh diff --git a/lib/check_public_network.sh b/lib/check_public_network.sh new file mode 100644 index 0000000..19a0f02 --- /dev/null +++ b/lib/check_public_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +public_network_ok() +{ + ping -c 1 -w 1 114.114.114.114 >/dev/null 2>&1 +} -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 12 Apr '21

12 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec.rb: delete submit_id when do submit spec
by Wei Jihui 12 Apr '21

12 Apr '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index bc411325..233aea26 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -3,7 +3,7 @@ require "#{LKP_SRC}/lib/yaml" def stable_yaml_file(yaml_file) job = load_yaml(yaml_file) - delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name) + delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name submit_id) job.delete_if do |key, _| delete_job_key.include?(key) || key.to_s.start_with?('#! ') || -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: set time when update job
by Wu Zhende 12 Apr '21

12 Apr '21

Can use time in kibana to limit the search scope Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/elasticsearch_client.cr | 3 +++ 2 files changed, 7 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8691082..b6bb730 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -315,6 +315,10 @@ class Job return package_dir end + def set_time + self["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def set_upload_dirs self["upload_dirs"] = "#{result_root}#{get_package_dir}" end diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 58bed08..0e83ff1 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -36,6 +36,9 @@ class Elasticsearch::Client # caller should judge response["_id"] != nil def set_job_content(job : Job) + # time indicates the update time of each job event + job.set_time + response = get_job_content(job.id) if response["id"]? response = update(job.dump_to_json_any, job.id) -- 2.23.0

1 0

[PATCH v1 compass-ci] extract-stats: add Fiber.yield after spawn operate
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] bin/es-jobs-mapping.sh: add time to mapping
by Wu Zhende 12 Apr '21

12 Apr '21

Add the time of the date type to the ES mapping. "time" indicates the last update time of the job. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 502e099..9926fd8 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -88,6 +88,9 @@ else "errid": { "type": "text" }, + "time": { + "type": "date" + }, "category": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix error: Missing hash key: "os_mount"
by Cao Xueliang 12 Apr '21

12 Apr '21

When you use os_mount in set os_dir, you should set os_mount at first. Reference: 23db97c43d4dd06d8c7edec50523a66b7b781a46 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index d3c5cf9..8691082 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -157,11 +157,11 @@ class Job append_init_field() set_docker_os() set_user_lkp_src() + set_os_mount() set_os_dir() set_submit_date() set_pp_params() set_rootfs() - set_os_mount() set_result_root() set_result_service() set_depends_initrd() -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add check_public_network.sh
by Liu Yinsi 12 Apr '21

12 Apr '21

Check public network ok or not during locally deploy Compass CI, because users need to sparrow in a pure intranet environment. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/check_public_network.sh | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 lib/check_public_network.sh diff --git a/lib/check_public_network.sh b/lib/check_public_network.sh new file mode 100644 index 0000000..2d60b31 --- /dev/null +++ b/lib/check_public_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +public_network_ok() +{ + ping -c 3 -w 1 114.114.114.114 >/dev/null 2>&1 +} -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: add Fiber.yield after spawn operation
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH lkp-tests] sshd: always exec sshd before start test
by Wei Jihui 12 Apr '21

12 Apr '21

submit -i ssh.yaml job.yaml before: run_test $LKP_SRC/tests/wrapper job start_daemon $LKP_SRC/daemon/sshd run_test $LKP_SRC/tests/wrapper sl now: run_setup $LKP_SRC/setup/sshd run_test $LKP_SRC/tests/wrapper job run_test $LKP_SRC/tests/wrapper sleep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- {daemon => setup}/sshd | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {daemon => setup}/sshd (100%) diff --git a/daemon/sshd b/setup/sshd similarity index 100% rename from daemon/sshd rename to setup/sshd -- 2.23.0

1 0

[PATCH compass-ci] sched: we can search the job in es before return job id in submit job
by Cao Xueliang 09 Apr '21

09 Apr '21

use refresh=wait_for to make sure the job can be searched before consume the job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/elasticsearch_client.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 3 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index f32256b..58bed08 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -142,6 +142,7 @@ class Elasticsearch::Client return @client.create( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => job_content, } @@ -152,6 +153,7 @@ class Elasticsearch::Client return @client.update( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => {:doc => job_content}, } diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 4f49394..41fa89f 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -76,6 +76,7 @@ class Sched if job_id begin job = @es.get_job(job_id.to_s) + @log.warn("job_is_nil, job id=#{job_id.to_s}") unless job rescue ex @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") @log.warn(ex.inspect_with_backtrace) -- 2.23.0

1 0

[PATCH v1 compass-ci 3/3] scheduler: add my start script for running myself scheduler
by Cao Xueliang 08 Apr '21

08 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/scheduler/my-start diff --git a/container/scheduler/my-start b/container/scheduler/my-start new file mode 100755 index 0000000..8d800f9 --- /dev/null +++ b/container/scheduler/my-start @@ -0,0 +1,85 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + REDIS_HOST + REDIS_PORT + ES_HOST + ES_PORT + OS_HTTP_HOST + OS_HTTP_PORT + INITRD_HTTP_HOST + INITRD_HTTP_PORT + TASKQUEUE_PORT + TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT + lab +] + +defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +env = docker_env(defaults) + +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] + +docker_rm "s001-alpine-#{SCHED_PORT}" + +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + +cmd = %W[ + docker run + --restart=always + --name s001-alpine-#{SCHED_PORT} + -d + -u 1090:1090 +] + bind_list + env + %W[ + -e CCI_REPOS=#{CCI_REPOS} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} + -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + -v /srv/initrd:/srv/initrd:ro + -v /srv/os:/srv/os:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + --log-opt tag=scheduler-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{SCHED_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 2/3] container: add scheduler-nginx for multiple scheduler nodes
by Cao Xueliang 08 Apr '21

08 Apr '21

use nginx as load balancing for multiple scheduler nodes. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler-nginx/Dockerfile | 8 +++++++ container/scheduler-nginx/build | 9 ++++++++ container/scheduler-nginx/nginx.conf | 29 +++++++++++++++++++++++++ container/scheduler-nginx/start | 32 ++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+) create mode 100644 container/scheduler-nginx/Dockerfile create mode 100755 container/scheduler-nginx/build create mode 100644 container/scheduler-nginx/nginx.conf create mode 100755 container/scheduler-nginx/start diff --git a/container/scheduler-nginx/Dockerfile b/container/scheduler-nginx/Dockerfile new file mode 100644 index 0000000..2cc430b --- /dev/null +++ b/container/scheduler-nginx/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM nginx:alpine + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +COPY nginx.conf /etc/nginx/ diff --git a/container/scheduler-nginx/build b/container/scheduler-nginx/build new file mode 100755 index 0000000..4b491d9 --- /dev/null +++ b/container/scheduler-nginx/build @@ -0,0 +1,9 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +docker_skip_rebuild "scheduler-nginx" + +docker build -t scheduler-nginx . diff --git a/container/scheduler-nginx/nginx.conf b/container/scheduler-nginx/nginx.conf new file mode 100644 index 0000000..c8c3783 --- /dev/null +++ b/container/scheduler-nginx/nginx.conf @@ -0,0 +1,29 @@ +# This is a default site configuration which will simply return 404, preventing +# chance access to any other virtualhost. +events { + worker_connections 65535; +} + +http{ + upstream sched { + server 172.17.0.1:3001; + server 172.17.0.1:3002; + server 172.17.0.1:3003; + } + + + server { + listen 3000 default_server; + listen [::]:3000 default_server; + + location / { + proxy_pass http://sched; + proxy_read_timeout 1814400; + } + + location = /404.html { + internal; + } + } +} + diff --git a/container/scheduler-nginx/start b/container/scheduler-nginx/start new file mode 100755 index 0000000..dc2dfbc --- /dev/null +++ b/container/scheduler-nginx/start @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_PORT +] + +defaults = relevant_defaults(names) +SCHED_PORT = defaults['SCHED_PORT'] +env = docker_env(defaults) + +docker_rm 'scheduler-nginx' + +cmd = %w[ + docker run + --name scheduler-nginx + --restart=always + -d +] + env + %W[ + -p #{SCHED_PORT}:#{SCHED_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + scheduler-nginx +] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 1/3] container: deploy three scheduler as the nginx upstream
by Cao Xueliang 08 Apr '21

08 Apr '21

use port 3001, 3002, 3003 as the scheduler port. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/start | 41 +++++++++++++++++++++++---------------- sparrow/3-code/dev-env | 7 +++++++ 2 files changed, 31 insertions(+), 17 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index 8d800f9..b13b396 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -7,8 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - SCHED_HOST SCHED_PORT + SCHED_NODES REDIS_HOST REDIS_PORT ES_HOST @@ -35,36 +35,42 @@ defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' -defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +sched_nodes = defaults.delete('SCHED_NODES') +SCHED_PORT = defaults.delete('SCHED_PORT') + +sched_nodes.each do |node| + docker_rm "s001-alpine-#{node['port']}" +end env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' -SCHED_PORT = defaults['SCHED_PORT'] -SCHED_HOST = defaults['SCHED_HOST'] -docker_rm "s001-alpine-#{SCHED_PORT}" +sched_nodes.each do |node| + node_host = node['host'] + node_port = node['port'] -bind_list = [] -[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| - bind_list << '-p' - bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" -end + bind_list = [] + [node_host, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{node_port}:#{node_port}" + end -cmd = %W[ + cmd = %W[ docker run - --restart=always - --name s001-alpine-#{SCHED_PORT} + --name s001-alpine-#{node_port} -d -u 1090:1090 -] + bind_list + env + %W[ + ] + bind_list + env + %W[ -e CCI_REPOS=#{CCI_REPOS} -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e SCHED_PORT=#{node_port} + -v /srv/tmp/public:/c/cci/scheduler/public -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} @@ -78,8 +84,9 @@ cmd = %W[ --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_PORT} sch-ruby-a:v0.00d-#{SCHED_PORT} -] + ] -cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + cmd += ['sh', '-c', 'umask 002 && ./scheduler'] -system(*cmd) + system(*cmd) +end diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 7327083..fd529e7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -77,6 +77,13 @@ ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port +SCHED_NODES: + - host: $sched_host + port: 3001 + - host: $sched_host + port: 3002 + - host: $sched_host + port: 3003 LKP_SERVER: $server_ip GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST -- 2.23.0

1 0

[PATCH compass-ci] scheduler: add my start script for running myself scheduler
by Cao Xueliang 08 Apr '21

08 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/my-start | 85 ++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/scheduler/my-start diff --git a/container/scheduler/my-start b/container/scheduler/my-start new file mode 100755 index 0000000..8d800f9 --- /dev/null +++ b/container/scheduler/my-start @@ -0,0 +1,85 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + REDIS_HOST + REDIS_PORT + ES_HOST + ES_PORT + OS_HTTP_HOST + OS_HTTP_PORT + INITRD_HTTP_HOST + INITRD_HTTP_PORT + TASKQUEUE_PORT + TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT + lab +] + +defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' +env = docker_env(defaults) + +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_PORT = defaults['SCHED_PORT'] +SCHED_HOST = defaults['SCHED_HOST'] + +docker_rm "s001-alpine-#{SCHED_PORT}" + +bind_list = [] +[SCHED_HOST, '172.17.0.1'].uniq.each do |ip| + bind_list << '-p' + bind_list << "#{ip}:#{SCHED_PORT}:#{SCHED_PORT}" +end + +cmd = %W[ + docker run + --restart=always + --name s001-alpine-#{SCHED_PORT} + -d + -u 1090:1090 +] + bind_list + env + %W[ + -e CCI_REPOS=#{CCI_REPOS} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} + -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + -v /srv/initrd:/srv/initrd:ro + -v /srv/os:/srv/os:ro + --log-opt mode=non-blocking + --log-opt max-buffer-size=4m + --log-opt tag=scheduler-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{SCHED_PORT} +] + +cmd += ['sh', '-c', 'umask 002 && ./scheduler'] + +system(*cmd) -- 2.23.0

1 0