- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..718b1fc --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + [[ -n "$my_email" && -n "$my_name" ]] && return + + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +export_config_variables + +[[ -n "$my_email" && -n "$my_name" ]] || { + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +} -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..77c4314 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd $CCI_SRC/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..f85c9ca63 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..da4dbce 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -26,9 +26,12 @@ class StatsWorker @tq.delete_task(queue_path + "/in_process", "#{job_id}") end rescue e - STDERR.puts e.message + error_message = e.message + STDERR.puts error_message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..62bb1aa --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + export_config_variables +fi + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +fi -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

2 2

[PATCH lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..c6e1cb450 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ - $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + $1.split.each do |repo| + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/lifecycle.cr: don't update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

Change the logic [Orginal] Update the testbox time in the ES database when reciving the job events [Now] Don't update in the ES. Only update the cache. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3a72906..16830d0 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -144,21 +144,17 @@ class Lifecycle def update_cached_machine(testbox, event) machine = @machines[testbox]? - return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + if machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) - update_es_machine_time(testbox, event) - end - - def update_es_machine_time(testbox, event) - machine = @es.get_tbox(testbox) - return unless machine - return unless event["time"].to_s.bigger_than?(machine["time"]?) + machine.as_h["time"] = event["time"] + else + machine = @es.get_tbox(testbox) + return unless machine - machine.as_h.delete("history") - machine.as_h["time"] = event["time"] - machine.as_h["state"] = JSON::Any.new("booting") - @machines[testbox] = machine - @es.update_tbox(testbox, machine.as_h) + machine.as_h.delete("history") + @machines[testbox] = machine + end end def update_cached_job(job_id, event) @@ -187,8 +183,7 @@ class Lifecycle def on_job_close(event) event_job_id = event["job_id"].to_s - job = @jobs[event_job_id] - + job = @jobs[event_job_id]? return unless job @jobs.delete(event_job_id) -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/scheduler: update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

When the machine accesses the API of the scheduler to update the job status, the machine's active time is also updated. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++++++ src/scheduler/update_job_parameter.cr | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51f493e..f5aba03 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -163,6 +163,13 @@ class Sched @es.update_tbox(testbox.to_s, hash) end + def update_testbox_time(job_id) + job = @redis.get_job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cc4cf87..b43aa06 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -9,6 +9,7 @@ class Sched end @env.set "job_id", job_id + @env.set "time", get_time # try to get report value and then update it job_content = {} of String => String @@ -34,13 +35,14 @@ class Sched @log.info(log.to_json) @env.set "job_state", job_content["job_state"]? + update_testbox_time(job_id) rescue e @log.warn(e) ensure mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => (@env.get?("job_state") || "update").to_s, - "time" => get_time + "time" => @env.get?("time").to_s } @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/0-package/read-config: export config yaml info
by Liu Yinsi 08 Mar '21

08 Mar '21

export my_email, my_name, server_ip to deploy compass-ci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..c664e30 --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,14 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +mkdir -p /etc/compass-ci/ +cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + +options=( server_ip my_name my_email ) + +for option in ${options[@]} +do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) +done -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 1

[PATCH compass-ci 1/5] container/dracut-initrd: add lkp module into conf file
by Xu Xijian 08 Mar '21

08 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/conf/add-modules.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/conf/add-modules.conf b/container/dracut-initrd/conf/add-modules.conf index b8048a1..6601277 100644 --- a/container/dracut-initrd/conf/add-modules.conf +++ b/container/dracut-initrd/conf/add-modules.conf @@ -1 +1 @@ -add_dracutmodules+=" cifs " +add_dracutmodules+=" cifs lkp " -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 08 Mar '21

08 Mar '21

move simplify-ci main function to $CCI_SRC/sparrow/install-client script, make code more reusablely. for simplicy-ci job test, if there is $SCHED_HOST, and export server_ip=$SCHED_HOST, when call $CCI_SRC/sparrow/install-client, no need to config sparrow/setup.yaml, and also no need to execute sparrow/0-package/read-config(export server_ip). for locally deploy compass-ci client, must config sparrow/setup.yaml, and also need to execute sparrow/0-package/read-config(export server_ip), if server_ip not exists, will call log_error() and exit. refer patch: Subject: [PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..107cef2e8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || return 1 +deploy -main -wait -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 08 Mar '21

08 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..5365bb2 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style - end - return nil unless pkg_style + next unless @hash[item]? - pkg_style = JSON.parse("{}") if pkg_style == nil + pkg_style.merge!((a)hash[item].as_h) + break unless pkg_style.empty? + end + return nil if pkg_style.empty? - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0

[PATCH lab-z9] jobs: add sysbench-mysql.yaml
by Wei Jihui 08 Mar '21

08 Mar '21

sysbench-mysql.yaml use lab-z9' cluster:cs-s1-a102-c1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 0000000..792645c --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,54 @@ +suite: sysbench-mysql +category: benchmark + +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: + +if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: NO_WAKEUP_PREEMPTION + iosched: deadline + iosched_dev: nvme0n1 + block_queue: + nr_requests: 2048 + block_queue_dev: nvme0n1 + nic_irp: + nics: enp189s0f1 + core_ranges: "55-79,93-95" + mysql: + data_dev: nvme0n1 + +if role client: + sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 2000000000 -- 2.23.0

1 0

[PATCH lkp-tests] tests/sysbench-mysql: accelerate data preparation
by Wei Jihui 08 Mar '21

08 Mar '21

1. accelerate data preparation from 20+min to 5+min 2. disable useless output Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 615bc6e7..0ade2dab 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,9 +1,6 @@ #!/bin/sh -# - oltp_test_mode # - oltp_tables_count # - oltp_table_size -# - max_requests -# - mysql_table_engine # - nr_threads # - rand_type # - rand_spec_pct @@ -21,14 +18,11 @@ : "${mysql_user:=root}" : "${mysql_host:=$server}" : "${mysql_port:=3306}" -: "${mysql_db:=sysbench_1}" +: "${mysql_db:=sysbench_test}" : "${mysql_password:=mysql_password}" : "${db_driver:=mysql}" -: "${oltp_test_mode:=complex}" : "${oltp_tables_count:=1000}" : "${oltp_table_size:=100000}" -: "${max_requests:=0}" -: "${mysql_table_engine:=innodb}" : "${rand_type:=special}" : "${rand_spec_pct:=100}" : "${nr_threads:=256}" @@ -42,13 +36,10 @@ args1=( --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db - --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver --mysql-password=$mysql_password - --max-requests=$max_requests - --mysql-table-engine=$mysql_table_engine - --oltp-table-size=$oltp_table_size - --oltp-tables-count=$oltp_tables_count + --table-size=$oltp_table_size + --tables=$oltp_tables_count --rand-type=$rand_type --rand-spec-pct=$rand_spec_pct --threads=$nr_threads @@ -72,8 +63,8 @@ args2=( stop_firewalld() { - systemctl stop firewalld - iptables -F + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 } run_sysbench_step() @@ -85,9 +76,11 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + sysbench ${args1[@]} oltp_read_write prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

1 0

[PATCH v3 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..279257cb6 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,24 @@ +# input: each line of test log with Array(String) structure which like: +# [ +# "time: 1600586885", +# "nr_free_pages 87512", +# ... +# "time: 1600586886", +# "nr_free_pages 87787", +# ... +# ] +# return eg: +# { +# "time"=>[1600586885, 1600586886], +# "nr_free_pages"=>[87512, 87787], +# ... +# } +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/dump-stat.rb: refactor lkp-tests/sbin/dump-stat
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] support handle hash from lkp-tests/stats/$script.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 194 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 lib/dump_stat.rb diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb new file mode 100644 index 0000000..1c0695e --- /dev/null +++ b/lib/dump_stat.rb @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: GPL-2.0-only + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/statistics" +require "#{LKP_SRC}/lib/bounds" +require "#{LKP_SRC}/lib/yaml" +require "#{LKP_SRC}/lib/job" +require "#{LKP_SRC}/lib/string_ext" +require "#{LKP_SRC}/lib/log" +require 'set' + +UNSTRUCTURED_MONITORS = %w[ftrace].to_set + +def warn_stat(msg, monitor) + log_warn msg + log_warn "check #{RESULT_ROOT}/#{monitor}" +end + +# dump stat which input by lkp-tests/stats/$script return +# input: +# eg-1: +# { +# "pgfree" => [275506, 280018], +# ... +# } +# eg-2: +# { +# "iperf.tcp.sender.bps" => 34804801216.197174, +# "iperf.tcp.receiver.bps" => "34804762215.18231" +# } +module DumpStat + def self.dump_stat(monitor, stat_result) + @result = {} + @invalid_records = [] + @record_index = 0 + @monitor = monitor + + stat_result.each do |key, value| + key = key.resolve_invalid_bytes + next if key[0] == '#' + next if value.empty? || value == 0 + next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ + + if key =~ /[ \t]/ + @invalid_records.push @record_index + warn_stat "whitespace in stats name: #{key}", @monitor + return nil # for exit current stats/script dump-stat + end + next if assign_log_message(key, value) + + k = @monitor + '.' + key + @result[k] ||= [] + fill_zero(k) + if value.is_a?(String) + value = check_string_value(k, value, @monitor) + next unless value + return nil unless number?(value, @invalid_records) + + value = value.index('.') ? value.to_f : value.to_i + elsif value.is_a?(Array) + (0..value.size - 1).each do |i| + next unless value[i].is_a?(String) + + value[i] = check_string_value(k, value[i], @monitor) + next unless value[i] + return nil unless number?(value[i], @invalid_records) + + value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i + valid_stats_verification(k, value[i]) + end + @result[k] = value + next + end + valid_stats_verification(k, value) + @result[k].push value + end + return nil if @result.empty? + + remove_zero_stats + delete_invalid_number(@result, @invalid_records, @monitor) + cols_verifation + return nil unless useful_result?(@result) + + save_json(@result, "#{RESULT_ROOT}/#{(a)monitor}.json", @result.size * @min_cols > 1000) + end + + # keep message | log line which key end with .message|.log + def self.assign_log_message(key, value) + if key.end_with?('.message', '.log') + k = @monitor + '.' + key + @result[k] = value + return true + end + + false + end + + def self.fill_zero(key) + size = @result[key].size + if @record_index < size + @record_index = size + elsif (@record_index - size).positive? + # fill 0 for missing values + @result[key].concat([0] * (@record_index - size)) + end + end + + def self.valid_stats_verification(key, value) + return nil if valid_stats_range? key, value + + @invalid_records.push @record_index + puts "outside valid range: #{value} in #{key} #{RESULT_ROOT}" + end + + def self.remove_zero_stats + @max_cols = 0 + @min_cols = Float::INFINITY + @min_cols_stat = '' + @max_cols_stat = '' + zero_stats = [] + @result.each do |key, val| + next if key.end_with?('.message', '.log') + + if @max_cols < val.size + @max_cols = val.size + @max_cols_stat = key + end + if @min_cols > val.size + @min_cols = val.size + @min_cols_stat = key + end + next if val[0] != 0 + next if val[-1] != 0 + next if val.sum != 0 + + zero_stats << key + end + zero_stats.each { |x| @result.delete x } + end + + def self.cols_verifation + return nil unless @min_cols < @max_cols && !UNSTRUCTURED_MONITORS.include?(@monitor) + + if @min_cols == @max_cols - 1 + @result.each { |_k, y| y.pop if y.size == @max_cols } + puts "Last record seems incomplete. Truncated #{RESULT_ROOT}/#{(a)monitor}.json" + else + warn_stat "Not a matrix: value size is different - #{@min_cols_stat}: #{@min_cols} != #{@max_cols_stat}: #{@max_cols}: #{RESULT_ROOT}/#{(a)monitor}.json", @monitor + end + end +end + +def check_string_value(key, value, monitor) + # value terminator is expected. If not, throw out an error warning. + warn_stat "no line terminator in stats value: #{value}", monitor if value.chomp!.nil? + + value.strip! + if value.empty? + warn_stat "empty stat value of #{key}", monitor + return nil + end + + return value +end + +# only number is valid +def number?(value, invalid_records) + unless value.numeric? + invalid_records.push record_index + warn_stat "invalid stats value: #{value}", monitor + return nil + end + + true +end + +def delete_invalid_number(result, invalid_records, monitor) + return nil if monitor == 'ftrace' + + invalid_records.reverse_each do |index| + result.each do |_k, value| + value.delete_at index + end + end +end + +def useful_result?(result) + return nil if result.empty? + return nil if result.values[0].size.zero? + return nil if result.values[-1].size.zero? + + true +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] old logical: lkp-tests/stats/srcipt < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/srcipt.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json [Dependence] lkp-tests/stats/$script.rb, for example: file_name: lkp-tests/stats/proc-vmstats.rb function name: proc_vmstats() input: Array(String) ["time: 231231234", "nr_free_page: 89721", ...] output: Hash(String, Array(Number|String) | Number | String) { "time" => [231231234, 231231235], "nr_free_page" => [89721,88564], ... } or { "time" => 231231234, "nr_free_page" => 89721, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 19 ++++++++++++++----- lib/stats_wrapper.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..94e370a 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,16 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file unless file.start_with?('.') + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +37,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +57,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..faa81df 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,27 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + log_lines = read_log("#{(a)log}.yaml") + stat_result = parse_simple_log_yaml(log_lines) + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +287,21 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end + +def parse_simple_log_yaml(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value + end + + result +end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 08 Mar '21

08 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..53c9e59 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -106,6 +107,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @log = JSONLogger.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") clone_upstream_repo load_fork_info @@ -192,6 +194,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +512,13 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + @log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..cd252faf1 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,10 @@ + +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] lib/json_logger.rb: fix that no warp cause no puts
by Li Yuanchao 05 Mar '21

05 Mar '21

The log didn't warp before, that caused sub-fluentd can not put the log out. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/json_logger.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 78f95ca..2c33c2f 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -29,7 +29,7 @@ class JSONLogger < Logger msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) - logger_hash.to_json + logger_hash.to_json + "\n" } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 05 Mar '21

05 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..ae7e64b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -192,6 +193,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +511,14 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + log = JSONLogger.new + log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v5 lkp-tests] daemon: add mysql
by Wei Jihui 05 Mar '21

05 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..808e3eec --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,91 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +prepare() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +insmod_getcpu() +{ + if [ -d "/getcpu" ]; then + cd /getcpu && insmod getcpu-hi.ko + else + echo "this test will not use getcpu optimize to bind core." + fi +} + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + prepare + insmod_getcpu + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0

[PATCH v5 compass-ci] doc/manual: add document for submit container
by Luan Shengde 05 Mar '21

05 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.en.md | 63 ++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/manual/build-lkp-tests-container.en.md diff --git a/doc/manual/build-lkp-tests-container.en.md b/doc/manual/build-lkp-tests-container.en.md new file mode 100644 index 0000000..47c85c5 --- /dev/null +++ b/doc/manual/build-lkp-tests-container.en.md @@ -0,0 +1,63 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need to install the lkp-tests to your local server. +Also you can avoid installation failures from undesired dependency package(s). + +# Prepare + + Do on your local server: + + - install docker + - apply account and config default yaml + - generate ssh key(s) + +# build container + +## 1. download resource + + Download lkp-tests and compass-ci to your local server. + + Command(s): + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. add environment variable(s) + + Command(s): + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build docker image + + Command(s): + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command(s): + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests on your own server. + It will start a disposable container to submit your job. + The container will attach the directory lkp-tests to the container itself. + You can edit the job yaml(s) in lkp-tests/jobs and it will take effect when you submit jobs. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + About submit: + + For detailed usage for command submit, please reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

1 0

[PATCH lkp-tests 08/11] install-iso-pre: add step: prepare_submit
by Yu Chuan 05 Mar '21

05 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 2 ++ tests/install-iso-pre | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre index 6f88eb919896..4569e1e3cdc0 100644 --- a/distro/depends/install-iso-pre +++ b/distro/depends/install-iso-pre @@ -1 +1,3 @@ lvm2 +ruby +git diff --git a/tests/install-iso-pre b/tests/install-iso-pre index e27c9ea80232..d867f1441003 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -163,7 +163,23 @@ prepare_ks() prepare_submit() { - : + ${LKP_SRC}/sbin/install-dependencies.sh + + export CCI_REPOS="/c" + + mkdir -p $CCI_REPOS && cd $CCI_REPOS + git clone $LAB_Z9_URL || die "clone lab-z9 git repo failed: $LAB_Z9_URL" + + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + mkdir -p $(dirname $config_yaml) + cat >> $config_yaml <<-EOF + SCHED_HOST: ${SCHED_HOST} + SCHED_PORT: ${SCHED_PORT} + my_name: ${my_name} + my_email: ${my_email} + my_token: ${MY_TOKEN} + lab: z9 + EOF } submit_install_iso_yaml() -- 2.23.0

1 0

[PATCH lkp-tests 01/11] feat: step 1 of iso2rootfs: install iso to testbox and reboot
by Yu Chuan 05 Mar '21

05 Mar '21

[Why] When testbox request job in pxe -> ipxe step, scheduler returns a response composed of ipxe cmd. So we add 'custom_ipxe' field of job.yaml, then we can customize the response what testbox get, then testbox can auto install iso by executing the specified ipxe cmd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/install-iso.yaml diff --git a/jobs/install-iso.yaml b/jobs/install-iso.yaml new file mode 100644 index 000000000000..bcccc6b76f85 --- /dev/null +++ b/jobs/install-iso.yaml @@ -0,0 +1,3 @@ +suite: install-iso + +custom_ipxe: -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 05 Mar '21

05 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..c07033c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -267,14 +267,19 @@ class Job pkg_style = JSON.parse("{}") if pkg_style == nil - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"]? || "#{os}" + tmp_os_arch = pkg_style["os_arch"]? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"]? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"]? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] jobs/iso2rootfs.yaml: change some field to secrets field.
by Li Ping 05 Mar '21

05 Mar '21

[why] the related fields when submit test.yaml should be add to secrets field. submit_name: submit_email: submit_token: and the required fields when submit test.yaml not have much to do with the iso2rootfs be changed to level-1 parameter. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/iso2rootfs.yaml | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55c4345fa..5df2cdf61 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -51,23 +51,24 @@ iso2rootfs: # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: - ################# - # submit test yaml related fields - ################# +################# +# submit test yaml related fields +################# - # three required fields when submit test yaml - submit_name: +# three required fields when submit test yaml +secrets: + submit_name: submit_email: submit_token: - # submit target tests yaml - ## 1. You can add as many jobs as you like. - ## 2. The following three fields is required for every test job. - test1_yaml: - test1_os_mount: - test1_testbox: - ## the following three fields is required when your job.yaml and - ## script for this test are from the internet. - test1_git_url: - test1_git_yaml: - test1_git_script: +# submit target tests yaml +## 1. You can add as many jobs as you like. +## 2. The following three fields is required for every test job. +test1_yaml: +test1_os_mount: +test1_testbox: +## the following three fields is required when your job.yaml and +## script for this test are from the internet. +test1_git_url: +test1_git_yaml: +test1_git_script: -- 2.23.0

1 1

[PATCH v2 lkp-tests 2/2] sbin/submit: optimize submit -r
by Li Ping 05 Mar '21

05 Mar '21

[why] submit -m -r iperf.yaml submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183160 submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183161 query=>{"job_id":["z9.1183160","z9.1183161"]} connect to ws://172.168.131.2:11310/filter {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:36.649+0800","job_id":"z9.1183161","message":"","job_state":"submit","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/udp-300/z9.1183161","status_code":200,"method":"POST","resource":"/submit_job","elapsed":"24.68ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.699+0800","job_id":"z9.1183160","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/tcp-300/z9.1183160","job_state":"set result root","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"8.75ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.735+0800","job_id":"z9.1183160","job_state":"boot","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"44.08ms"} ==> only get one result [how] get result_root directly from the msg not from fluentd data Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/monitor.rb | 36 ++++++++++-------------------------- sbin/monitor | 8 ++++++++ sbin/submit | 4 +++- 3 files changed, 21 insertions(+), 27 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 78582be6a..24191fa8d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :stop_query + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :result_roots, :stop_query def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,9 +25,9 @@ class Monitor @exit_status_code = 0 @defaults = {} @result = [] + @result_roots = [] @stop_query = {} @reason = nil - @result_root = nil end def load_default @@ -95,29 +95,13 @@ class Monitor exec cmd end - def set_result_root(data) - return unless @action['mirror_result'] - return unless data['log'] - - data = JSON.parse(data['log']) - return unless data['result_root'] - - @result_root = data['result_root'] - end - - def lftp_mirror - @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" - system "lftp -c mirror #{url} >/dev/null 2>&1" - end - - def mirror_result(data) - if @result_root - lftp_mirror - else - set_result_root(data) + def mirror_result + @result_roots.each do |res| + res.to_s.delete_prefix!('/srv') + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_port = job['SRV_HTTP_PORT'] || '11300' + url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + system "lftp -c mirror #{url} >/dev/null 2>&1" end end @@ -163,7 +147,7 @@ class Monitor output(data) connect(data, ws) - mirror_result(data) + mirror_result stop(data, ws) if @action['stop'] end diff --git a/sbin/monitor b/sbin/monitor index def47bf6a..a2e677fc0 100755 --- a/sbin/monitor +++ b/sbin/monitor @@ -12,6 +12,7 @@ require 'yaml' opt_set_key_value = {} opt_action = {'output' => true} opt_job = {} +opt_results = [] opt_monitor_url = nil opt_timeout = 0 opt_stop_query = {} @@ -35,6 +36,12 @@ options = OptionParser.new do |opts| opt_action[action] = true end end + + opts.on('-r result_roots', '--results results', 'set monitor result_roots') do |result_roots| + result_roots.split(',').each do |res| + opt_results << res + end + end opts.separator ' action: output, output log' opts.separator ' action: connect, connect to the host when get ip' opts.separator ' action: stop, exit monitor client when get log' @@ -79,6 +86,7 @@ monitor = Monitor.new monitor.monitor_url = opt_monitor_url if opt_monitor_url monitor.overrides = opt_set_key_value monitor.action = opt_action +monitor.result_roots = opt_results monitor.job = opt_job monitor.stop_query = opt_stop_query diff --git a/sbin/submit b/sbin/submit index a6f0cd387..44fd0a7db 100755 --- a/sbin/submit +++ b/sbin/submit @@ -19,6 +19,7 @@ opt_monitor = false opt_monitor_query = {} opt_my_queue = false actions = ['output', 'stop'] +result_roots = [] options = OptionParser.new do |opts| opts.banner = 'Usage: submit [options] job1.yaml job2.yaml ...' @@ -197,6 +198,7 @@ ARGV.each do |jobfile| messages = scheduler_client.submit_job(job_json) JSON.parse(messages).each do |msg| if msg['message'].empty? + result_roots << msg['result_root'] job_ids << msg['job_id'].to_s puts("submit #{jobfile}, got job id=#{msg['job_id']}") else @@ -210,6 +212,6 @@ end if opt_monitor job_hash_list[0].delete('define_files') opt_monitor_query.merge!({'job_id' => job_ids}) - cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -s 'job_state: extract_finished'" + cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -r #{result_roots.join(',')} -s 'job_state: extract_finished'" exec cmd end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: fix cmdline parameter parse float
by Wei Jihui 05 Mar '21

05 Mar '21

[error] input: submit job.yaml a=6.10 output at job.yaml: a: 6.1 [why] Yaml.load will load 6.10 as float to 6.1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sbin/submit b/sbin/submit index a6f0cd38..e5e41828 100755 --- a/sbin/submit +++ b/sbin/submit @@ -28,7 +28,8 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on("-s 'KEY: VALUE'", "--set 'KEY: VALUE'", 'add YAML hash to job') do |key_value| - opt_set_key_value.merge! YAML.load key_value + k, v = key_value.sub(' ', '').split(':') + opt_set_key_value[k] = v end opts.on('-o DIR', '--output DIR', 'save job yaml to DIR/') do |dir| @@ -58,9 +59,9 @@ options = OptionParser.new do |opts| opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do opt_monitor = true - filter = YAML.load ARGV[0] if ARGV[0] - if filter.class == Hash - opt_monitor_query.merge! filter + k, v = ARGV[0].sub(' ', '').split(':') if ARGV[0] + if (k && !k.empty?) && (v && !v.empty?) + opt_monitor_query[k] = v ARGV.shift end end @@ -77,10 +78,11 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' + k, v = arg.sub(' ', '').sub(/=/, ':').split(':') if seen_job_yaml - opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') + opt_set_key_value[k] = v else - opt_cmdline_defaults.merge! YAML.load arg.sub(/=/, ': ') + opt_cmdline_defaults[k] = v end true else -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: optimize submit -r
by Li Ping 05 Mar '21

05 Mar '21

[why] submit -m -r iperf.yaml submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183160 submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183161 query=>{"job_id":["z9.1183160","z9.1183161"]} connect to ws://172.168.131.2:11310/filter {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:36.649+0800","job_id":"z9.1183161","message":"","job_state":"submit","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/udp-300/z9.1183161","status_code":200,"method":"POST","resource":"/submit_job","elapsed":"24.68ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.699+0800","job_id":"z9.1183160","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/tcp-300/z9.1183160","job_state":"set result root","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"8.75ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.735+0800","job_id":"z9.1183160","job_state":"boot","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"44.08ms"} ==> only get one result [how] get result_root directly from the msg not from fluentd data Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/monitor.rb | 36 ++++++++++-------------------------- sbin/monitor | 8 ++++++++ sbin/submit | 4 +++- 3 files changed, 21 insertions(+), 27 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 78582be6a..996bec749 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :stop_query + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :result_roots, :stop_query def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,9 +25,9 @@ class Monitor @exit_status_code = 0 @defaults = {} @result = [] + @result_roots = result_roots @stop_query = {} @reason = nil - @result_root = nil end def load_default @@ -95,29 +95,13 @@ class Monitor exec cmd end - def set_result_root(data) - return unless @action['mirror_result'] - return unless data['log'] - - data = JSON.parse(data['log']) - return unless data['result_root'] - - @result_root = data['result_root'] - end - - def lftp_mirror - @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" - system "lftp -c mirror #{url} >/dev/null 2>&1" - end - - def mirror_result(data) - if @result_root - lftp_mirror - else - set_result_root(data) + def mirror_result + @result_roots.each do |res| + res.to_s.delete_prefix!('/srv') + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_port = job['SRV_HTTP_PORT'] || '11300' + url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + system "lftp -c mirror #{url} >/dev/null 2>&1" end end @@ -163,7 +147,7 @@ class Monitor output(data) connect(data, ws) - mirror_result(data) + mirror_result stop(data, ws) if @action['stop'] end diff --git a/sbin/monitor b/sbin/monitor index def47bf6a..a2e677fc0 100755 --- a/sbin/monitor +++ b/sbin/monitor @@ -12,6 +12,7 @@ require 'yaml' opt_set_key_value = {} opt_action = {'output' => true} opt_job = {} +opt_results = [] opt_monitor_url = nil opt_timeout = 0 opt_stop_query = {} @@ -35,6 +36,12 @@ options = OptionParser.new do |opts| opt_action[action] = true end end + + opts.on('-r result_roots', '--results results', 'set monitor result_roots') do |result_roots| + result_roots.split(',').each do |res| + opt_results << res + end + end opts.separator ' action: output, output log' opts.separator ' action: connect, connect to the host when get ip' opts.separator ' action: stop, exit monitor client when get log' @@ -79,6 +86,7 @@ monitor = Monitor.new monitor.monitor_url = opt_monitor_url if opt_monitor_url monitor.overrides = opt_set_key_value monitor.action = opt_action +monitor.result_roots = opt_results monitor.job = opt_job monitor.stop_query = opt_stop_query diff --git a/sbin/submit b/sbin/submit index a6f0cd387..44fd0a7db 100755 --- a/sbin/submit +++ b/sbin/submit @@ -19,6 +19,7 @@ opt_monitor = false opt_monitor_query = {} opt_my_queue = false actions = ['output', 'stop'] +result_roots = [] options = OptionParser.new do |opts| opts.banner = 'Usage: submit [options] job1.yaml job2.yaml ...' @@ -197,6 +198,7 @@ ARGV.each do |jobfile| messages = scheduler_client.submit_job(job_json) JSON.parse(messages).each do |msg| if msg['message'].empty? + result_roots << msg['result_root'] job_ids << msg['job_id'].to_s puts("submit #{jobfile}, got job id=#{msg['job_id']}") else @@ -210,6 +212,6 @@ end if opt_monitor job_hash_list[0].delete('define_files') opt_monitor_query.merge!({'job_id' => job_ids}) - cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -s 'job_state: extract_finished'" + cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -r #{result_roots.join(',')} -s 'job_state: extract_finished'" exec cmd end -- 2.23.0

1 0

[PATCH lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 05 Mar '21

05 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..d49002801 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,10 @@ + +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split + result[key.chomp(':')] << value.to_i + end + + result +end -- 2.23.0

2 3

[PATCH lkp-tests] add test script for dockerfile os migrating
by Bai Jing 05 Mar '21

05 Mar '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 139 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..b77b890a9 --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,139 @@ +#!/bin/sh +# - dockerfile_url + +# exit 1 if openEuler failed +# exit 0 if +# - no centos support +# - centos failed + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty" + +cd $BENCHMARK_ROOT + +centos_one_ok=false + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || { + echo "current-repo not support centos" + exit 0 + } +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + i=1 + for file in $(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]' | awk -F: '{print $1}') + do + file_dir=${file%/*} + image=$(echo "$file" | awk -F '/' '{print $3}' | tr 'A-Z' 'a-z') + [ -n "$image" ] || image=${file#*/}_$i + if [ -f "$file_dir/build.sh" ] && [ -x "$file_dir/build.sh" ]; then + "$file_dir/build.sh" > /dev/null 2> /dev/null && { + centos_one_ok=true + handle_dockerfile + } + else + docker_build > /dev/null 2> /dev/null && { + centos_one_ok=true + handle_dockerfile + } + fi + let i++ + done + [ "$centos_one_ok" == "false" ] && { + echo "centos all failed" + exit 0 + } +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$file" + +} + +handle_epel() +{ + grep -qw "epel-release" "$file" && { + sed -i "s/epel-release/bash/g" "$file" + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" "$file" && { + sed -i "s|https://.*epel-release.*rpm|bash|g" "$file" + + } +} + +handle_groupadd() +{ + grep -qw "groupadd*" "$file" && { + sed -i '/groupadd*/i\ yum -y install shadow && \\' "$file" + } + +} + +handle_epel_repo() +{ + grep -qwE '.* wget .*(epel|CentOS-Base)\.repo' "$file" && { + sed 's|^|#|g' "$file" + } +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + handle_groupadd + docker_build "-openeuler" + [ $? -ne 0 ] || docker_run "-openeuler" +} + +docker_build() +{ + docker build -f "$file" -t "${image}$1" . + [ $? -ne 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

2 2

[PATCH v2 compass-ci] service/lifecycle: add lifecycle
by Wu Zhende 05 Mar '21

05 Mar '21

Function: processing timeout jobs/machines processing crash jobs/machines Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 317 +++++++++++++++++++++++++- src/lifecycle.cr | 17 +- src/lifecycle/constants.cr | 3 + src/lifecycle/lifecycle.cr | 19 +- src/scheduler/elasticsearch_client.cr | 18 +- 5 files changed, 339 insertions(+), 35 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index af5cd07..3a72906 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -1,19 +1,43 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +require "set" require "kemal" require "yaml" -require "./web_env" +require "./mq" +require "./scheduler_api" require "../scheduler/elasticsearch_client" +require "../lifecycle/constants" + +class String + def bigger_than?(time) + return false if self.empty? + + time = time.to_s + return true if time.empty? + + # Historical data contains time in the format of "%Y-%m-%d %H:%M:%S" + # Compatibility processing is required + time = time.gsub(/ /, "T") + time = Time.parse(time, "%Y-%m-%dT%H:%M:%S", Time.local.location) + self_time = Time.parse(self, "%Y-%m-%dT%H:%M:%S", Time.local.location) + + self_time > time + end +end class Lifecycle property es - def initialize(env : HTTP::Server::Context) + def initialize + @mq = MQClient.instance @es = Elasticsearch::Client.new - @env = env - @log = env.log.as(JSONLogger) + @scheduler_api = SchedulerAPI.new + @log = JSONLogger.new + @jobs = Hash(String, JSON::Any).new + @machines = Hash(String, JSON::Any).new + @match = Hash(String, Set(String)).new {|h, k| h[k] = Set(String).new} end def alive(version) @@ -22,18 +46,291 @@ class Lifecycle @log.warn(e) end - def get_running_testbox - size = @env.params.query["size"]? || 20 - from = @env.params.query["from"]? || 0 + def init_from_es + jobs = get_active_jobs + jobs.each do |result| + job_id = result["_id"].to_s + job = result["_source"].as_h + job.delete_if{|key, _| !JOB_KEYWORDS.includes?(key)} + + @jobs[job_id] = JSON.parse(job.to_json) + @match[job["testbox"].to_s] << job_id + end + + machines = get_active_machines + machines.each do |result| + testbox = result["_id"].to_s + machine = result["_source"].as_h + machine.delete("history") + + machine = JSON.parse(machine.to_json) + @machines[testbox] = machine + + deal_match_job(testbox, machine["job_id"].to_s) + end + end + + def deal_match_job(testbox, job_id) + @match[testbox].each do |id| + next if id == job_id + + msg = { + "job_id" => id, + "job_state" => "abnormal", + "testbox" => testbox + } + @mq.pushlish_confirm("job_mq", msg.to_json) + @match[testbox].delete(id) + end + end + + def get_active_jobs query = { - "size" => size, - "from" => from, + "size" => 10000, + "query" => { + "term" => { + "job_state" => "boot" + } + } + } + @es.search("jobs", query) + end + + def get_active_machines + query = { + "size" => 10000, "query" => { "terms" => { - "state" => ["booting", "running"] + "state" => ["booting", "running", "rebooting"] } } } @es.search("testbox", query) end + + def mq_event_loop + puts "deal job events" + q = @mq.ch.queue("job_mq") + q.subscribe(no_ack: false) do |msg| + event = JSON.parse(msg.body_io.to_s) + job_state = event["job_state"]? + + case job_state + when "boot" + on_job_boot(event) + when "close" + on_job_close(event) + when "abnormal" + on_abnormal_job(event) + else + on_other_job(event) + end + @mq.ch.basic_ack(msg.delivery_tag) + end + end + + def on_other_job(event) + event_job_id = event["job_id"].to_s + return if event_job_id.empty? + + update_cached_job(event_job_id, event) + + job = @jobs[event_job_id]? + return unless job + + testbox = job["testbox"].to_s + update_cached_machine(testbox, event) + end + + def update_cached_machine(testbox, event) + machine = @machines[testbox]? + return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + + update_es_machine_time(testbox, event) + end + + def update_es_machine_time(testbox, event) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + machine.as_h.delete("history") + machine.as_h["time"] = event["time"] + machine.as_h["state"] = JSON::Any.new("booting") + @machines[testbox] = machine + @es.update_tbox(testbox, machine.as_h) + end + + def update_cached_job(job_id, event) + job = @jobs[job_id]? + if job + @jobs[job_id] = JSON.parse(job.as_h.merge!(event.as_h).to_json) + else + job = @es.get_job(job_id) + return unless job + return if JOB_CLOSE_STATE.includes?(job["job_state"]?) + + job = job.dump_to_json_any.as_h + job.delete_if{|key, _| !JOB_KEYWORDS.includes?(key)} + job["job_state"] = event["job_state"] + @jobs[job_id] = JSON.parse(job.to_json) + end + end + + def on_abnormal_job(event) + event_job_id = event["job_id"].to_s + return unless @jobs.has_key?(event_job_id) + + @jobs.delete(event_job_id) + spawn @scheduler_api.close_job(event_job_id, "abnormal", "lifecycle") + end + + def on_job_close(event) + event_job_id = event["job_id"].to_s + job = @jobs[event_job_id] + + return unless job + + @jobs.delete(event_job_id) + update_cached_machine(job["testbox"].to_s, event) + end + + def on_job_boot(event) + event_job_id = event["job_id"]?.to_s + @jobs[event_job_id] = event unless event_job_id.empty? + machine_info = @machines[event["testbox"]]? + deal_boot_machine(machine_info, event) + end + + def deal_boot_machine(machine_info, event) + event_job_id = event["job_id"]?.to_s + + unless machine_info + @machines[event["testbox"].to_s] = event + return + end + + machine_job_id = machine_info["job_id"].to_s + # The job is not updated + # No action is required + return if event_job_id == machine_job_id + + time = machine_info["time"]? + # Skip obsolete event + return unless event["time"].to_s.bigger_than?(time) + + @machines[event["testbox"].to_s] = event + + deal_match_job(event["testbox"].to_s, event_job_id) + + # No previous job to process + return if machine_job_id.empty? + return unless @jobs.has_key?(machine_job_id) + + @jobs.delete(machine_job_id) + spawn @scheduler_api.close_job(machine_job_id, "abnormal", "lifecycle") + end + + def max_time(times) + result = "" + times.each do |time| + result = time if time.to_s.bigger_than?(result) + end + return result + end + + def timeout_job_loop + dead_job_id = nil + loop do + close_job(dead_job_id, "timeout") if dead_job_id + deadline, dead_job_id = get_min_deadline + # deal timeout job + next if dead_job_id && deadline <= Time.local + + sleep_until(deadline) + end + end + + def timeout_machine_loop + dead_machine_name = nil + loop do + reboot_timeout_machine(dead_machine_name) if dead_machine_name + deadline, dead_machine_name = get_min_deadline_machine + next if dead_machine_name && deadline <= Time.local + + sleep_until(deadline) + end + end + + def sleep_until(deadline) + s = (deadline - Time.local).total_seconds + sleep(s) + end + + def get_min_deadline + deadline = (Time.local + 60.second) + dead_job_id = nil + @jobs.each do |id, job| + next unless job["deadline"]? + job_deadline = Time.parse(job["deadline"].to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) + return job_deadline, id if Time.local >= job_deadline + next unless deadline > job_deadline + + deadline = job_deadline + dead_job_id = id + end + return deadline, dead_job_id + end + + def get_min_deadline_machine + deadline = (Time.local + 60.second) + dead_machine_name = nil + @machines.each do |name, machine| + next if machine["deadline"]?.to_s.empty? + + machine_deadline = Time.parse(machine["deadline"].to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) + return machine_deadline, name if Time.local >= machine_deadline + next unless deadline > machine_deadline + + deadline = machine_deadline + dead_machine_name = name + end + return deadline, dead_machine_name + end + + def close_job(job_id, reason) + @jobs.delete(job_id) + spawn @scheduler_api.close_job(job_id, reason, "lifecycle") + end + + def reboot_timeout_machine(testbox) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return if MACHINE_CLOSE_STATE.includes?(machine["state"]) + + deadline = machine["deadline"]? + return unless deadline + + deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) + return if Time.local < deadline + + mq_queue = get_machine_reboot_queue(testbox) + @mq.pushlish_confirm(mq_queue, machine.to_json) + + machine["state"] = "rebooting_queue" + machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + @es.update_tbox(testbox, machine.as_h) + end + + def get_machine_reboot_queue(testbox) + if testbox.includes?(".") + testbox =~ /.*\.(.*)-\d+$/ + else + testbox =~ /(.*)--.*/ + end + $1 + rescue + testbox + end end diff --git a/src/lifecycle.cr b/src/lifecycle.cr index a864621..f6b80a8 100644 --- a/src/lifecycle.cr +++ b/src/lifecycle.cr @@ -2,15 +2,20 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. require "lifecycle/lifecycle" -require "./lifecycle/constants.cr" +require "./lifecycle/constants" require "./lib/json_logger" +require "./lib/lifecycle" module Cycle log = JSONLogger.new + lifecycle = Lifecycle.new - begin - Kemal.run(LIFECYCLE_PORT) - rescue e - log.error(e) - end + # init @jobs and @machines + lifecycle.init_from_es + lifecycle.mq_event_loop + + spawn lifecycle.timeout_job_loop + spawn lifecycle.timeout_machine_loop + + Kemal.run(LIFECYCLE_PORT) end diff --git a/src/lifecycle/constants.cr b/src/lifecycle/constants.cr index 137b7f9..694a407 100644 --- a/src/lifecycle/constants.cr +++ b/src/lifecycle/constants.cr @@ -2,3 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. LIFECYCLE_PORT = (ENV.has_key?("LIFECYCLE_PORT") ? ENV["LIFECYCLE_PORT"] : 11312).to_i32 +JOB_CLOSE_STATE = ["abnormal", "close", "failed", "finished", "timeout", "crash"] +JOB_KEYWORDS = ["testbox", "job_state", "deadline"] +MACHINE_CLOSE_STATE = ["real_rebooting", "rebooting_queue"] diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr index da42fda..c318721 100644 --- a/src/lifecycle/lifecycle.cr +++ b/src/lifecycle/lifecycle.cr @@ -9,24 +9,7 @@ require "../lib/json_logger" module Cycle VERSION = "0.1.0" - - add_context_storage_type(Time::Span) - - before_all do |env| - env.set "start_time", Time.monotonic - env.response.headers["Connection"] = "close" - env.create_log - env.create_lifecycle - end - - # echo alive get "/" do |env| - env.lifecycle.alive(VERSION) - end - - # find the testbox that are performing jobs - # curl http://localhost:11312/get_running_testbox?size=10&from=0 - get "/get_running_testbox" do |env| - env.lifecycle.get_running_testbox.to_json + "done" end end diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 91fb84a..f6ff618 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -79,7 +79,10 @@ class Elasticsearch::Client end def search(index, query) - @client.search({:index => index, :body => query}) + results = @client.search({:index => index, :body => query}) + raise results unless results.is_a?(JSON::Any) + + results["hits"]["hits"].as_a end def update_account(account_content : JSON::Any, my_email : String) @@ -92,6 +95,19 @@ class Elasticsearch::Client ) end + def get_tbox(testbox) + query = {:index => "testbox", :type => "_doc", :id => testbox} + return nil unless @client.exists(query) + + response = @client.get_source(query) + case response + when JSON::Any + return response + else + return nil + end + end + def update_tbox(testbox : String, wtmp_hash : Hash) query = {:index => "testbox", :type => "_doc", :id => testbox} if @client.exists(query) -- 2.23.0

1 0

[PATCH v4 compass-ci] container/submit: add attach lkp-tests
by Luan Shengde 05 Mar '21

05 Mar '21

add attach lkp-tests to the container when run the container to submit jobs. [why] enable user edit code or job file(s). user can edit the code or job file(s) according to their requires. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/container/submit/submit b/container/submit/submit index 6236892..724e04e 100755 --- a/container/submit/submit +++ b/container/submit/submit @@ -13,6 +13,7 @@ cmd=( --name=submit-$USER-$data_suffix -it -v /etc/compass-ci:/etc/compass-ci:ro + -v $LKP_SRC:/srv/lkp-tests:ro -v $HOME/.config:/root/.config:ro -v $HOME/.ssh:/root/.ssh:rw submit -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 05 Mar '21

05 Mar '21

[Why] old logical: lkp-tests/stats/srcipt < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/srcipt.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 19 ++++++++++++++----- lib/stats_wrapper.rb | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..94e370a 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,16 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file unless file.start_with?('.') + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +37,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +57,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..09662be 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,26 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + stat_result = YAML.safe_load("#{(a)log}.yaml") + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +286,11 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 04 Mar '21

04 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..a30c969 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -274,7 +274,11 @@ class Job tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

2 2

[PATCH compass-ci] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 04 Mar '21

04 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..92a3830 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -192,6 +192,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs) end def do_push(fork_key) @@ -509,4 +510,15 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, new_refs) + log = { + msg: 'new refs', + date: Date.today, + time: Time.now, + repo: git_repo, + nr_new_branch: new_refs[:heads].length + }.to_json + puts log + end end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit
by Wang Chenglong 04 Mar '21

04 Mar '21

[Why] In 'job.yaml The 'os_version' should use quotation marks around the value, converted to a string type. Otherwise we use 'os_version: 6.10', The submit will get error. wcl@crystal ~/my_yaml% cat iperf.yaml os: centos os_arch: aarch64 os_version: 6.10 os_mount: cifs wcl@crystal ~/my_yaml% submit iperf.yaml submit_id=983c3605-03e2-4e8f-8ac4-c78325463fde submit iperf.yaml failed, got job id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/6.1/boot/vmlinuz': No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a6f0cd387..993e27f96 100755 --- a/sbin/submit +++ b/sbin/submit @@ -147,6 +147,10 @@ ARGV.each do |jobfile| jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] jobs['submit_id'] = submit_id + jobs['os_version'].is_a?(String) || ( + puts "Please use quotation marks around value: os_version: \'#{jobs['os_version']}\'" + exit + ) jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

2 1

help star
by Wu Fengguang 04 Mar '21

04 Mar '21

Jihui is our help star, thanks Jihui! ~/todo/bin% ./help-star 21.6 Wu Fengguang 13.2 Wei Jihui 12.2 Cao Xueliang 11.0 Wu Zhende 5.8 Lin Jiaxin 5.6 Luan Shengde 5.3 Liu Yinsi 5.1 Lu Weitao 4.5 Yu Chuan 4.5 Wang Yong 3.7 Li Ping 2.5 Zhang Dewan 2.2 Cui Yili 1.8 Wang Chenglong 1.1 Bai Jing 0.0 Tong Qunfeng 0.0 Xu Xijian 0.0 Hu Xuejiao 0.0 Du Kaitian 0.0 Liu Shaofei 0.0 Xiao Shenwei 0.0 Zhang Yale 0.0 Li Yuanchao 0.0 Ren Wen 0.0 Wang Shuai 0.0 Lu Kaiyi 0.0 Cui Xucui

1 0

git count
by Wu Fengguang 04 Mar '21

04 Mar '21

Xueliang, Zhende is our code star, thanks Xueliang, Zhende! wfg@crystal ~/todo/bin% ./git-count 8 Cao Xueliang 8 Wu Zhende 6 Liu Yinsi 6 Wei Jihui 4 Yu Chuan 4 Wang Yong 4 Li Yuanchao 3 Lu Weitao 2 Wang Chenglong 2 Luan Shengde 2 Lin Jiaxin 0 Bai Jing 0 Tong Qunfeng 0 Xu Xijian 0 Hu Xuejiao 0 Du Kaitian 0 Liu Shaofei 0 Sun Yukui 0 Wu Fengguang 0 Xiao Shenwei 0 Zhang Yale 0 Zhang Yu 0 Cui Yili 0 Ren Wen 0 Zhang Dewan 0 Li Ping 0 Lu Kaiyi 0 Shi Zhichao w

1 0

review star
by Wu Fengguang 04 Mar '21

04 Mar '21

Jihui is our review star again, thanks Jihui! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 41 22 45 20 Wu Fengguang 61 18 27 15 Wei Jihui 68 9 14 12 Cao Xueliang 41 14 36 10 Luan Shengde 23 0 0 3 Yu Chuan 13 2 2 2 Wang Yong 15 1 2 2 Wang Chenglong 2 1 0 1 Bai Jing 9 0 0 0 Wu Zhende 30 0 0 0 Liu Yinsi 13 0 0 0 Li Yuanchao 13 0 0 0 Lu Weitao 14 0 0 0 Xu Xijian 12 0 0 0 Li Ping 5 1 1 0 Lin Jiaxin 13 0 0 -1 Cui Yili

1 0

[PATCH v3 compass-ci] container/submit: add attach lkp-tests
by Luan Shengde 04 Mar '21

04 Mar '21

add attach lkp-tests to the container when run the container to submit jobs. [why] enable user edit code or job file(s). user can edit the code or job file(s) according to their requires. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 1 + 1 file changed, 1 insertion(+) diff --git a/container/submit/submit b/container/submit/submit index 6236892..ada05f9 100755 --- a/container/submit/submit +++ b/container/submit/submit @@ -13,6 +13,7 @@ cmd=( --name=submit-$USER-$data_suffix -it -v /etc/compass-ci:/etc/compass-ci:ro + -v $LKP_SRC:/root/lkp-tests:ro -v $HOME/.config:/root/.config:ro -v $HOME/.ssh:/root/.ssh:rw submit -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: improve bisect report content
by Cao Xueliang 04 Mar '21

04 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 7ce5660..782c702 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -54,7 +54,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - We found some error/warning(s) and the first bad commit in the below project: + Some error/warning(s) are found in git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/dev-env: expose the SRV_HTTP host and port for job
by Cao Xueliang 04 Mar '21

04 Mar '21

Reference: 36a7848fe733b36a5b75c81560b32b400c4c1b98 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index df7b1ce..403ccb1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -84,6 +84,12 @@ LKP_SERVER: GIT_SERVER: GITCACHE_HOST: GITCACHE_PORT: +SRV_HTTP_RESULT_HOST: +SRV_HTTP_RESULT_PORT: +SRV_HTTP_OS_HOST: +SRV_HTTP_OS_PORT: +SRV_HTTP_GIT_HOST: +SRV_HTTP_GIT_PORT: SRV_HTTP_CCI_HOST: SRV_HTTP_CCI_PORT: SEND_MAIL_HOST: -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/iso2rootfs.yaml: change some field to secrets field.
by Li Ping 04 Mar '21

04 Mar '21

[why] the related fields when submit test.yaml should be add to secrets field. submit_name: submit_email: submit_token: and the required fields when submit test.yaml not have much to do with the iso2rootfs be changed to level-1 parameter. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/iso2rootfs.yaml | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55c4345fa..b2b4a5520 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -51,23 +51,24 @@ iso2rootfs: # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: - ################# - # submit test yaml related fields - ################# +################# +# submit test yaml related fields +################# - # three required fields when submit test yaml +# three required fields when submit test yaml +secrets: submit_name: submit_email: submit_token: - # submit target tests yaml - ## 1. You can add as many jobs as you like. - ## 2. The following three fields is required for every test job. - test1_yaml: - test1_os_mount: - test1_testbox: - ## the following three fields is required when your job.yaml and - ## script for this test are from the internet. - test1_git_url: - test1_git_yaml: - test1_git_script: +# submit target tests yaml +## 1. You can add as many jobs as you like. +## 2. The following three fields is required for every test job. +test1_yaml: +test1_os_mount: +test1_testbox: +## the following three fields is required when your job.yaml and +## script for this test are from the internet. +test1_git_url: +test1_git_yaml: +test1_git_script: -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: use new SRV_HTTP host and port for result and git
by Cao Xueliang 04 Mar '21

04 Mar '21

Reference: 36a7848fe733b36a5b75c81560b32b400c4c1b98 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 503ed43..7ce5660 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -40,11 +40,12 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" - bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" + prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv_result}#{ENV['result_root']}\n" : '' report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" - pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv_git}/git/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv_result}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} -- 2.23.0

1 1

[PATCH v2 compass-ci] lib/mail_bisect_result: use new SRV_HTTP port for result and git
by Cao Xueliang 04 Mar '21

04 Mar '21

Reference: 36a7848fe733b36a5b75c81560b32b400c4c1b98 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 503ed43..7ce5660 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -40,11 +40,12 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" - bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" + prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv_result}#{ENV['result_root']}\n" : '' report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" - pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv_git}/git/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv_result}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} -- 2.23.0

1 0