- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] sparrow/install-tiny: skip steps to deploy compass-ci server
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: execute "./install-tiny", locally deploy compass-ci, can submit job, run containers services, run qemu test. after: execute "./install-tiny server reboot", locally deploy compass-ci server, just run containers servers. [why] some user who locally deploy compass-ci, the host which just run containers can only access intranet, so skip some steps that need to access internet when deploy, can submit job and run qemu test will in other machine, use $LKP_SRC/setup/simply-ci to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index cb47cf5..414c5bc 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -4,10 +4,19 @@ # For your reference only. # It's better to run the below scripts step by step. +export server=$1 +export action=$2 + +server_skip() +{ + local deploy_script=$1 + [ "$server" != "server" ] && ${deploy_script} +} + [[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) cd $CCI_SRC/sparrow || exit -0-package/install +server_skip 0-package/install 0-package/common 1-storage/tiny 1-storage/permission @@ -15,9 +24,9 @@ cd $CCI_SRC/sparrow || exit 2-network/iptables 2-network/nfs 2-network/cifs -3-code/git +server_skip 3-code/git 3-code/dev-env . /etc/profile.d/compass.sh -4-docker/buildall -5-build/ipxe +4-docker/buildall $action +server_skip 5-build/ipxe 7-systemd/systemd-setup -- 2.23.0

3 7

[PATCH v3 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sbin: add script for download os rootfs
by Liu Yinsi 29 Jan '21

29 Jan '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100644 index 0000000..d5ea328 --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs + +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +export os=$1 +export os_arch=$2 +export os_version=$3 + +: ${os:=openeuler} +: ${os_arch:=aarch64} +: ${os_version:=20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 1

[PATCH v3 lkp-tests] setup/simplify-ci: add param $server_ip
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: all containers services ip is the host locally server ip after: if $server_ip is not null, all containers services ip is $server_ip [why] if container services deploy in other server, add param $server_ip will easy to deploy, rather than change config yaml manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..4b26e3bcb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,7 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/submit: README file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..9ce7c25 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,30 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit almost all the linux OS. +# Avoid much repetitive testing work to adapt. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Edit executable file: + + submit + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your localhost. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/submit: submit file
by Luan Shengde 29 Jan '21

29 Jan '21

executable file for container: subject it can be run as the real command submit, when you run it, it will calls the command 'submit' in the container add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..20fdf61 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/submit: build file
by Luan Shengde 29 Jan '21

29 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 29 Jan '21

29 Jan '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..fa5509d --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0

[PATCH compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 29 Jan '21

29 Jan '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

1 0