March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/submit: add attach directory
by Luan Shengde 03 Mar '21

03 Mar '21

add attach lkp-tests/hosts to the container when run the container [why] enable user submit jobs with custom job files. user can edit the job files according their requires, and the job file will take effect when use submit contanier to submit jobs. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/container/submit/submit b/container/submit/submit index 6236892..ca6eb72 100755 --- a/container/submit/submit +++ b/container/submit/submit @@ -13,6 +13,7 @@ cmd=( --name=submit-$USER-$data_suffix -it -v /etc/compass-ci:/etc/compass-ci:ro + -v $LKP_SRC/jobs:/root/lkp-tests/jobs:ro -v $HOME/.config:/root/.config:ro -v $HOME/.ssh:/root/.ssh:rw submit -- 2.23.0

2 2

[PATCH compass-ci 2/4] sparrow/0-package/read-config: export config yaml info
by Liu Yinsi 03 Mar '21

03 Mar '21

export my_email, my_name, server_ip to deploy compass-ci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..9da6ffc --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,14 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +mkdir -p /etc/compass-ci/ +cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + +options=( server_ip my_name my_email ) + +for option in ${options[@]} +do + export $option=$(grep "^$option:" /etc/compass-ci/setup.yaml |awk -F ": " '{print $2}') +done -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 03 Mar '21

03 Mar '21

move simplify-ci main function to $CCI_SRC/sparrow/install-client script, make code more reusablely. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..107cef2e8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || return 1 +deploy -main -wait -- 2.23.0

2 2

[PATCH v2 compass-ci] doc/manual: add document for submit container
by Luan Shengde 03 Mar '21

03 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-test-container.en.md | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/manual/build-lkp-test-container.en.md diff --git a/doc/manual/build-lkp-test-container.en.md b/doc/manual/build-lkp-test-container.en.md new file mode 100644 index 0000000..d021185 --- /dev/null +++ b/doc/manual/build-lkp-test-container.en.md @@ -0,0 +1,57 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need install the lkp-tests to your own server. +Also you can avoid installation failures for undesired dependency packages. + +# Prepare + +- install docker +- apply account and config default yaml +- generate ssh keys + +# build submit container + +## 1. download resource + + Use the following command to downloac lkp-test and compass-ci + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. setup environment variables + + Command: + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build submit image + + Command: + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command: + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests at your own server. + It will start a disposable container to submit your job. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + About summit: + + For detailed usage for command submit, reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

2 2

[PATCH v3 compass-ci] doc/manual: add document for submit container
by Luan Shengde 03 Mar '21

03 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-test-container.en.md | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/manual/build-lkp-test-container.en.md diff --git a/doc/manual/build-lkp-test-container.en.md b/doc/manual/build-lkp-test-container.en.md new file mode 100644 index 0000000..2852a49 --- /dev/null +++ b/doc/manual/build-lkp-test-container.en.md @@ -0,0 +1,57 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need install the lkp-tests to your own server. +Also you can avoid installation failures for undesired dependency packages. + +# Prepare + +- install docker +- apply account and config default yaml +- generate ssh keys + +# build submit container + +## 1. download resource + + Use the following command to download lkp-test and compass-ci + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. setup environment variables + + Command: + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build submit image + + Command: + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command: + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests on your own server. + It will start a disposable container to submit your job. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + About summit: + + For detailed usage for command submit, please reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 03 Mar '21

03 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..aa48bce 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,9 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + container_id=$(docker ps -aqf name="ssh_r") + [ -n "$container_id" ] && exit + [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 1

[PATCH compass-ci] lib/stats_wrapper.rb: remove delete tmpfile
by Lu Weitao 03 Mar '21

03 Mar '21

[Why] the tmpfile will be deleted automatically with the object reclaimed by GC Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..d4ac6e2 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -227,8 +227,6 @@ module StatsWrapper return nil end - File.delete(file) - true end -- 2.23.0

1 1

[PATCH compass-ci 2/2] lib/scheduler_api.cr: add parameters
by Wu Zhende 03 Mar '21

03 Mar '21

Add job_state to indicates the status of a customized job. Add source to specify source. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr index b8dd514..50dc495 100644 --- a/src/lib/scheduler_api.cr +++ b/src/lib/scheduler_api.cr @@ -10,9 +10,11 @@ class SchedulerAPI @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" end - def close_job(job_id) + def close_job(job_id, job_state = nil, source=nil) + url = "/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}&source=#{source}" + url += "&job_state=#{job_state}" if job_state client = HTTP::Client.new(@host, port: @port) - response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + response = client.get(url) client.close() return response end -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/scheduler: send mq message
by Wu Zhende 03 Mar '21

03 Mar '21

When the scheduler service receives a job event, sends a message to the MQ queue. The lifecycle service receives and processes these events. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 ++ src/scheduler/close_job.cr | 13 +++++++++++++ src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 13 +++++++++++++ src/scheduler/update_job_parameter.cr | 11 +++++++++++ 5 files changed, 41 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 5f28de9..bec90cd 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -4,6 +4,7 @@ require "kemal" require "yaml" +require "./mq" require "./job" require "./web_env" require "./block_helper" @@ -34,6 +35,7 @@ class Sched @es = Elasticsearch::Client.new Redis::Client.set_pool_size(1000) @redis = Redis::Client.instance + @mq = MQClient.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new @env = env diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index b3e6716..ea1dbf5 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -6,11 +6,14 @@ class Sched job_id = @env.params.query["job_id"]? return unless job_id + @env.set "job_id", job_id + job = @redis.get_job(job_id) # update job_state job_state = @env.params.query["job_state"]? job["job_state"] = job_state if job_state + job["job_state"] = "complete" if job["job_state"] == "boot" response = @es.set_job_content(job) if response["_id"] == nil @@ -34,5 +37,15 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) + ensure + source = @env.params.query["source"]? + if source != "lifecycle" + mq_msg = { + "job_id" => @env.get?("job_id").to_s, + "job_state" => "close", + "time" => get_time + } + @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + end end end diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index d7d7a9a..3fdb983 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -12,6 +12,8 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" +JOB_MQ = "job_mq" + LAB = (ENV.has_key?("lab") ? ENV["lab"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b7fe0e7..a876abc 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -16,6 +16,7 @@ class Sched host = value end + @env.set "testbox", host response = get_job_boot(host, boot_type) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id @@ -23,6 +24,15 @@ class Sched response rescue e @log.warn(e) + ensure + mq_msg = { + "job_id" => @env.get?("job_id").to_s, + "testbox" => @env.get?("testbox").to_s, + "deadline" => @env.get?("deadline").to_s, + "time" => get_time, + "job_state" => "boot" + } + @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end # auto submit a job to collect the host information. @@ -104,6 +114,9 @@ class Sched if job @es.set_job_content(job) + @env.set "job_id", job["id"] + @env.set "deadline", job["deadline"] + @env.set "job_state", job["job_state"] create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) else # for physical machines diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 744274a..cc4cf87 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -8,6 +8,8 @@ class Sched return false end + @env.set "job_id", job_id + # try to get report value and then update it job_content = {} of String => String job_content["id"] = job_id @@ -30,7 +32,16 @@ class Sched log = job_content.dup log["job_id"] = log.delete("id").not_nil! @log.info(log.to_json) + + @env.set "job_state", job_content["job_state"]? rescue e @log.warn(e) + ensure + mq_msg = { + "job_id" => @env.get?("job_id").to_s, + "job_state" => (@env.get?("job_state") || "update").to_s, + "time" => get_time + } + @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: add document for submit container
by Luan Shengde 03 Mar '21

03 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-test-container.en.md | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/manual/build-lkp-test-container.en.md diff --git a/doc/manual/build-lkp-test-container.en.md b/doc/manual/build-lkp-test-container.en.md new file mode 100644 index 0000000..b94fcc3 --- /dev/null +++ b/doc/manual/build-lkp-test-container.en.md @@ -0,0 +1,57 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need install the lkp-tests to your own server. +Also you can avoid installation failures for undesired dependency packages. + +# Prepare + +- install docker +- apply account and config default yaml +- generate ssh keys + +# build submit container + +## 1. download resource + + Use the following command to downloac lkp-test and compass-ci + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. setup environment variables + + Command: + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build submit image + + Command: + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command: + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests at your own server. + It will start a disposable contanier to submit your job. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + Command helps: + + For detailed usage for command submit, reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

2 2