- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..718b1fc --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + [[ -n "$my_email" && -n "$my_name" ]] && return + + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +export_config_variables + +[[ -n "$my_email" && -n "$my_name" ]] || { + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +} -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..77c4314 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd $CCI_SRC/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..f85c9ca63 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci] /src/extract-stats: rescue when the task-queue disconnect
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] the connection between stats_worker and task-queue may be interrupted, and the old connection will be disable, so stats_worker need create a new connection with task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 836d1f8..da4dbce 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -26,9 +26,12 @@ class StatsWorker @tq.delete_task(queue_path + "/in_process", "#{job_id}") end rescue e - STDERR.puts e.message + error_message = e.message + STDERR.puts error_message + # incase of many error message when task-queue, ES does not work sleep(10) + @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") end end end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/0-package/read-config: export config yaml variables
by Liu Yinsi 08 Mar '21

08 Mar '21

if ENV $my_email or $my_name doesn't exist, export config yaml variables firstly, if not config sparrow/setup.yaml, call log_error() and exit. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..62bb1aa --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +. $CCI_SRC/lib/log.sh + +export_config_variables() +{ + mkdir -p /etc/compass-ci/ + cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + + options=( server_ip my_name my_email ) + + for option in ${options[@]} + do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) + done +} + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + export_config_variables +fi + +if [ -z "$my_email" ] || [ -z "$my_name" ]; then + log_error "ENV \$my_email or \$my_name doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 +fi -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

2 2

[PATCH lkp-tests] stats: add stats script for dockerfile migrating
by Bai Jing 08 Mar '21

08 Mar '21

[input] Error: Unable to find a match: supervisor /bin/sh: tar: command not found [output] sh.command-not-found.tar: 1 yum.error.Unable-to-find-a-match.supervisor: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index 749f3667b..c6e1cb450 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -1,12 +1,43 @@ #!/usr/bin/env ruby +require 'set' + +repo_set = Set[] +sys_set = Set[] while (line = STDIN.gets) case line.chomp! + # Error: Unable to find a match: docker-registry when /Error: Unable to find a match: (.+)/ - $1.split.each do |repo| - puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + $1.split.each do |repo| + repo_set << repo end + + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" + puts "sh.command-not-found.#{$1}.message: #{line}" + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + puts "requested-URL-returned.error: 1" + puts "requested-URL-returned.error.message: #{line}" end end + +repo_set.each do |repo| + puts "yum.error.Unable-to-find-a-match.#{repo}: 1" + puts "yum.error.Unable-to-find-a-match.#{repo}.message: Error: Unable to find a match: #{repo}" +end + +sys_set.each do |sys| + puts "yum.swap.error.unrecognized-arguments-install.#{sys}: 1" + puts "yum.swap.error.unrecognized-arguments.#{sys}.message: yum swap: error: unrecognized arguments: install #{sys}" +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/lifecycle.cr: don't update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

Change the logic [Orginal] Update the testbox time in the ES database when reciving the job events [Now] Don't update in the ES. Only update the cache. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3a72906..16830d0 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -144,21 +144,17 @@ class Lifecycle def update_cached_machine(testbox, event) machine = @machines[testbox]? - return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + if machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) - update_es_machine_time(testbox, event) - end - - def update_es_machine_time(testbox, event) - machine = @es.get_tbox(testbox) - return unless machine - return unless event["time"].to_s.bigger_than?(machine["time"]?) + machine.as_h["time"] = event["time"] + else + machine = @es.get_tbox(testbox) + return unless machine - machine.as_h.delete("history") - machine.as_h["time"] = event["time"] - machine.as_h["state"] = JSON::Any.new("booting") - @machines[testbox] = machine - @es.update_tbox(testbox, machine.as_h) + machine.as_h.delete("history") + @machines[testbox] = machine + end end def update_cached_job(job_id, event) @@ -187,8 +183,7 @@ class Lifecycle def on_job_close(event) event_job_id = event["job_id"].to_s - job = @jobs[event_job_id] - + job = @jobs[event_job_id]? return unless job @jobs.delete(event_job_id) -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/scheduler: update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

When the machine accesses the API of the scheduler to update the job status, the machine's active time is also updated. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++++++ src/scheduler/update_job_parameter.cr | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51f493e..f5aba03 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -163,6 +163,13 @@ class Sched @es.update_tbox(testbox.to_s, hash) end + def update_testbox_time(job_id) + job = @redis.get_job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cc4cf87..b43aa06 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -9,6 +9,7 @@ class Sched end @env.set "job_id", job_id + @env.set "time", get_time # try to get report value and then update it job_content = {} of String => String @@ -34,13 +35,14 @@ class Sched @log.info(log.to_json) @env.set "job_state", job_content["job_state"]? + update_testbox_time(job_id) rescue e @log.warn(e) ensure mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => (@env.get?("job_state") || "update").to_s, - "time" => get_time + "time" => @env.get?("time").to_s } @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/0-package/read-config: export config yaml info
by Liu Yinsi 08 Mar '21

08 Mar '21

export my_email, my_name, server_ip to deploy compass-ci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..c664e30 --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,14 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +mkdir -p /etc/compass-ci/ +cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + +options=( server_ip my_name my_email ) + +for option in ${options[@]} +do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) +done -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 1

[PATCH compass-ci 1/5] container/dracut-initrd: add lkp module into conf file
by Xu Xijian 08 Mar '21

08 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/conf/add-modules.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/conf/add-modules.conf b/container/dracut-initrd/conf/add-modules.conf index b8048a1..6601277 100644 --- a/container/dracut-initrd/conf/add-modules.conf +++ b/container/dracut-initrd/conf/add-modules.conf @@ -1 +1 @@ -add_dracutmodules+=" cifs " +add_dracutmodules+=" cifs lkp " -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 08 Mar '21

08 Mar '21

move simplify-ci main function to $CCI_SRC/sparrow/install-client script, make code more reusablely. for simplicy-ci job test, if there is $SCHED_HOST, and export server_ip=$SCHED_HOST, when call $CCI_SRC/sparrow/install-client, no need to config sparrow/setup.yaml, and also no need to execute sparrow/0-package/read-config(export server_ip). for locally deploy compass-ci client, must config sparrow/setup.yaml, and also need to execute sparrow/0-package/read-config(export server_ip), if server_ip not exists, will call log_error() and exit. refer patch: Subject: [PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..107cef2e8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || return 1 +deploy -main -wait -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 08 Mar '21

08 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..5365bb2 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style - end - return nil unless pkg_style + next unless @hash[item]? - pkg_style = JSON.parse("{}") if pkg_style == nil + pkg_style.merge!((a)hash[item].as_h) + break unless pkg_style.empty? + end + return nil if pkg_style.empty? - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0

[PATCH lab-z9] jobs: add sysbench-mysql.yaml
by Wei Jihui 08 Mar '21

08 Mar '21

sysbench-mysql.yaml use lab-z9' cluster:cs-s1-a102-c1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 0000000..792645c --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,54 @@ +suite: sysbench-mysql +category: benchmark + +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: + +if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: NO_WAKEUP_PREEMPTION + iosched: deadline + iosched_dev: nvme0n1 + block_queue: + nr_requests: 2048 + block_queue_dev: nvme0n1 + nic_irp: + nics: enp189s0f1 + core_ranges: "55-79,93-95" + mysql: + data_dev: nvme0n1 + +if role client: + sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 2000000000 -- 2.23.0

1 0

[PATCH lkp-tests] tests/sysbench-mysql: accelerate data preparation
by Wei Jihui 08 Mar '21

08 Mar '21

1. accelerate data preparation from 20+min to 5+min 2. disable useless output Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 615bc6e7..0ade2dab 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,9 +1,6 @@ #!/bin/sh -# - oltp_test_mode # - oltp_tables_count # - oltp_table_size -# - max_requests -# - mysql_table_engine # - nr_threads # - rand_type # - rand_spec_pct @@ -21,14 +18,11 @@ : "${mysql_user:=root}" : "${mysql_host:=$server}" : "${mysql_port:=3306}" -: "${mysql_db:=sysbench_1}" +: "${mysql_db:=sysbench_test}" : "${mysql_password:=mysql_password}" : "${db_driver:=mysql}" -: "${oltp_test_mode:=complex}" : "${oltp_tables_count:=1000}" : "${oltp_table_size:=100000}" -: "${max_requests:=0}" -: "${mysql_table_engine:=innodb}" : "${rand_type:=special}" : "${rand_spec_pct:=100}" : "${nr_threads:=256}" @@ -42,13 +36,10 @@ args1=( --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db - --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver --mysql-password=$mysql_password - --max-requests=$max_requests - --mysql-table-engine=$mysql_table_engine - --oltp-table-size=$oltp_table_size - --oltp-tables-count=$oltp_tables_count + --table-size=$oltp_table_size + --tables=$oltp_tables_count --rand-type=$rand_type --rand-spec-pct=$rand_spec_pct --threads=$nr_threads @@ -72,8 +63,8 @@ args2=( stop_firewalld() { - systemctl stop firewalld - iptables -F + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 } run_sysbench_step() @@ -85,9 +76,11 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + sysbench ${args1[@]} oltp_read_write prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

1 0

[PATCH v3 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..279257cb6 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,24 @@ +# input: each line of test log with Array(String) structure which like: +# [ +# "time: 1600586885", +# "nr_free_pages 87512", +# ... +# "time: 1600586886", +# "nr_free_pages 87787", +# ... +# ] +# return eg: +# { +# "time"=>[1600586885, 1600586886], +# "nr_free_pages"=>[87512, 87787], +# ... +# } +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/dump-stat.rb: refactor lkp-tests/sbin/dump-stat
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] support handle hash from lkp-tests/stats/$script.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 194 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 lib/dump_stat.rb diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb new file mode 100644 index 0000000..1c0695e --- /dev/null +++ b/lib/dump_stat.rb @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: GPL-2.0-only + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/statistics" +require "#{LKP_SRC}/lib/bounds" +require "#{LKP_SRC}/lib/yaml" +require "#{LKP_SRC}/lib/job" +require "#{LKP_SRC}/lib/string_ext" +require "#{LKP_SRC}/lib/log" +require 'set' + +UNSTRUCTURED_MONITORS = %w[ftrace].to_set + +def warn_stat(msg, monitor) + log_warn msg + log_warn "check #{RESULT_ROOT}/#{monitor}" +end + +# dump stat which input by lkp-tests/stats/$script return +# input: +# eg-1: +# { +# "pgfree" => [275506, 280018], +# ... +# } +# eg-2: +# { +# "iperf.tcp.sender.bps" => 34804801216.197174, +# "iperf.tcp.receiver.bps" => "34804762215.18231" +# } +module DumpStat + def self.dump_stat(monitor, stat_result) + @result = {} + @invalid_records = [] + @record_index = 0 + @monitor = monitor + + stat_result.each do |key, value| + key = key.resolve_invalid_bytes + next if key[0] == '#' + next if value.empty? || value == 0 + next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ + + if key =~ /[ \t]/ + @invalid_records.push @record_index + warn_stat "whitespace in stats name: #{key}", @monitor + return nil # for exit current stats/script dump-stat + end + next if assign_log_message(key, value) + + k = @monitor + '.' + key + @result[k] ||= [] + fill_zero(k) + if value.is_a?(String) + value = check_string_value(k, value, @monitor) + next unless value + return nil unless number?(value, @invalid_records) + + value = value.index('.') ? value.to_f : value.to_i + elsif value.is_a?(Array) + (0..value.size - 1).each do |i| + next unless value[i].is_a?(String) + + value[i] = check_string_value(k, value[i], @monitor) + next unless value[i] + return nil unless number?(value[i], @invalid_records) + + value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i + valid_stats_verification(k, value[i]) + end + @result[k] = value + next + end + valid_stats_verification(k, value) + @result[k].push value + end + return nil if @result.empty? + + remove_zero_stats + delete_invalid_number(@result, @invalid_records, @monitor) + cols_verifation + return nil unless useful_result?(@result) + + save_json(@result, "#{RESULT_ROOT}/#{(a)monitor}.json", @result.size * @min_cols > 1000) + end + + # keep message | log line which key end with .message|.log + def self.assign_log_message(key, value) + if key.end_with?('.message', '.log') + k = @monitor + '.' + key + @result[k] = value + return true + end + + false + end + + def self.fill_zero(key) + size = @result[key].size + if @record_index < size + @record_index = size + elsif (@record_index - size).positive? + # fill 0 for missing values + @result[key].concat([0] * (@record_index - size)) + end + end + + def self.valid_stats_verification(key, value) + return nil if valid_stats_range? key, value + + @invalid_records.push @record_index + puts "outside valid range: #{value} in #{key} #{RESULT_ROOT}" + end + + def self.remove_zero_stats + @max_cols = 0 + @min_cols = Float::INFINITY + @min_cols_stat = '' + @max_cols_stat = '' + zero_stats = [] + @result.each do |key, val| + next if key.end_with?('.message', '.log') + + if @max_cols < val.size + @max_cols = val.size + @max_cols_stat = key + end + if @min_cols > val.size + @min_cols = val.size + @min_cols_stat = key + end + next if val[0] != 0 + next if val[-1] != 0 + next if val.sum != 0 + + zero_stats << key + end + zero_stats.each { |x| @result.delete x } + end + + def self.cols_verifation + return nil unless @min_cols < @max_cols && !UNSTRUCTURED_MONITORS.include?(@monitor) + + if @min_cols == @max_cols - 1 + @result.each { |_k, y| y.pop if y.size == @max_cols } + puts "Last record seems incomplete. Truncated #{RESULT_ROOT}/#{(a)monitor}.json" + else + warn_stat "Not a matrix: value size is different - #{@min_cols_stat}: #{@min_cols} != #{@max_cols_stat}: #{@max_cols}: #{RESULT_ROOT}/#{(a)monitor}.json", @monitor + end + end +end + +def check_string_value(key, value, monitor) + # value terminator is expected. If not, throw out an error warning. + warn_stat "no line terminator in stats value: #{value}", monitor if value.chomp!.nil? + + value.strip! + if value.empty? + warn_stat "empty stat value of #{key}", monitor + return nil + end + + return value +end + +# only number is valid +def number?(value, invalid_records) + unless value.numeric? + invalid_records.push record_index + warn_stat "invalid stats value: #{value}", monitor + return nil + end + + true +end + +def delete_invalid_number(result, invalid_records, monitor) + return nil if monitor == 'ftrace' + + invalid_records.reverse_each do |index| + result.each do |_k, value| + value.delete_at index + end + end +end + +def useful_result?(result) + return nil if result.empty? + return nil if result.values[0].size.zero? + return nil if result.values[-1].size.zero? + + true +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] old logical: lkp-tests/stats/srcipt < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/srcipt.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json [Dependence] lkp-tests/stats/$script.rb, for example: file_name: lkp-tests/stats/proc-vmstats.rb function name: proc_vmstats() input: Array(String) ["time: 231231234", "nr_free_page: 89721", ...] output: Hash(String, Array(Number|String) | Number | String) { "time" => [231231234, 231231235], "nr_free_page" => [89721,88564], ... } or { "time" => 231231234, "nr_free_page" => 89721, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 19 ++++++++++++++----- lib/stats_wrapper.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..94e370a 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,16 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file unless file.start_with?('.') + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +37,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +57,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..faa81df 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,27 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + log_lines = read_log("#{(a)log}.yaml") + stat_result = parse_simple_log_yaml(log_lines) + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +287,21 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end + +def parse_simple_log_yaml(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value + end + + result +end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 08 Mar '21

08 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..53c9e59 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -106,6 +107,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @log = JSONLogger.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") clone_upstream_repo load_fork_info @@ -192,6 +194,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +512,13 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + @log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..cd252faf1 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,10 @@ + +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] lib/json_logger.rb: fix that no warp cause no puts
by Li Yuanchao 05 Mar '21

05 Mar '21

The log didn't warp before, that caused sub-fluentd can not put the log out. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/json_logger.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 78f95ca..2c33c2f 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -29,7 +29,7 @@ class JSONLogger < Logger msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) - logger_hash.to_json + logger_hash.to_json + "\n" } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 05 Mar '21

05 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..ae7e64b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -192,6 +193,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +511,14 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + log = JSONLogger.new + log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v5 lkp-tests] daemon: add mysql
by Wei Jihui 05 Mar '21

05 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..808e3eec --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,91 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +prepare() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +insmod_getcpu() +{ + if [ -d "/getcpu" ]; then + cd /getcpu && insmod getcpu-hi.ko + else + echo "this test will not use getcpu optimize to bind core." + fi +} + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + prepare + insmod_getcpu + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0