- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..718b1fc --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + [[ -n "$my_email" && -n "$my_name" ]] && return + + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +export_config_variables + +[[ -n "$my_email" && -n "$my_name" ]] || { + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +} -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..77c4314 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd $CCI_SRC/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..f85c9ca63 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..da4dbce 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -26,9 +26,12 @@ class StatsWorker @tq.delete_task(queue_path + "/in_process", "#{job_id}") end rescue e - STDERR.puts e.message + error_message = e.message + STDERR.puts error_message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..62bb1aa --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + export_config_variables +fi + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +fi -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

2 2

[PATCH lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..c6e1cb450 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ - $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + $1.split.each do |repo| + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/lifecycle.cr: don't update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

Change the logic [Orginal] Update the testbox time in the ES database when reciving the job events [Now] Don't update in the ES. Only update the cache. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3a72906..16830d0 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -144,21 +144,17 @@ class Lifecycle def update_cached_machine(testbox, event) machine = @machines[testbox]? - return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + if machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) - update_es_machine_time(testbox, event) - end - - def update_es_machine_time(testbox, event) - machine = @es.get_tbox(testbox) - return unless machine - return unless event["time"].to_s.bigger_than?(machine["time"]?) + machine.as_h["time"] = event["time"] + else + machine = @es.get_tbox(testbox) + return unless machine - machine.as_h.delete("history") - machine.as_h["time"] = event["time"] - machine.as_h["state"] = JSON::Any.new("booting") - @machines[testbox] = machine - @es.update_tbox(testbox, machine.as_h) + machine.as_h.delete("history") + @machines[testbox] = machine + end end def update_cached_job(job_id, event) @@ -187,8 +183,7 @@ class Lifecycle def on_job_close(event) event_job_id = event["job_id"].to_s - job = @jobs[event_job_id] - + job = @jobs[event_job_id]? return unless job @jobs.delete(event_job_id) -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/scheduler: update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

When the machine accesses the API of the scheduler to update the job status, the machine's active time is also updated. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++++++ src/scheduler/update_job_parameter.cr | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51f493e..f5aba03 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -163,6 +163,13 @@ class Sched @es.update_tbox(testbox.to_s, hash) end + def update_testbox_time(job_id) + job = @redis.get_job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cc4cf87..b43aa06 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -9,6 +9,7 @@ class Sched end @env.set "job_id", job_id + @env.set "time", get_time # try to get report value and then update it job_content = {} of String => String @@ -34,13 +35,14 @@ class Sched @log.info(log.to_json) @env.set "job_state", job_content["job_state"]? + update_testbox_time(job_id) rescue e @log.warn(e) ensure mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => (@env.get?("job_state") || "update").to_s, - "time" => get_time + "time" => @env.get?("time").to_s } @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end -- 2.23.0

1 0