- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..36a70fd --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +[[ -n "$my_email" && -n "$my_name" ]] && return +export_config_variables + +[[ -n "$my_email" && -n "$my_name" ]] && return +log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" +exit 1 -- 2.23.0

2 2

[PATCH v5 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..718b1fc --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + [[ -n "$my_email" && -n "$my_name" ]] && return + + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +export_config_variables + +[[ -n "$my_email" && -n "$my_name" ]] || { + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +} -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..77c4314 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd $CCI_SRC/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..f85c9ca63 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..da4dbce 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -26,9 +26,12 @@ class StatsWorker @tq.delete_task(queue_path + "/in_process", "#{job_id}") end rescue e - STDERR.puts e.message + error_message = e.message + STDERR.puts error_message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..62bb1aa --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + export_config_variables +fi + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +fi -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

2 2

[PATCH lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..c6e1cb450 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ - $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + $1.split.each do |repo| + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/lifecycle.cr: don't update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

Change the logic [Orginal] Update the testbox time in the ES database when reciving the job events [Now] Don't update in the ES. Only update the cache. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3a72906..16830d0 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -144,21 +144,17 @@ class Lifecycle def update_cached_machine(testbox, event) machine = @machines[testbox]? - return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + if machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) - update_es_machine_time(testbox, event) - end - - def update_es_machine_time(testbox, event) - machine = @es.get_tbox(testbox) - return unless machine - return unless event["time"].to_s.bigger_than?(machine["time"]?) + machine.as_h["time"] = event["time"] + else + machine = @es.get_tbox(testbox) + return unless machine - machine.as_h.delete("history") - machine.as_h["time"] = event["time"] - machine.as_h["state"] = JSON::Any.new("booting") - @machines[testbox] = machine - @es.update_tbox(testbox, machine.as_h) + machine.as_h.delete("history") + @machines[testbox] = machine + end end def update_cached_job(job_id, event) @@ -187,8 +183,7 @@ class Lifecycle def on_job_close(event) event_job_id = event["job_id"].to_s - job = @jobs[event_job_id] - + job = @jobs[event_job_id]? return unless job @jobs.delete(event_job_id) -- 2.23.0

1 0