- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/compare_matrixes.rb: fix spell error
by Lu Kaiyi 01 Dec '20

01 Dec '20

there should be stddev_percent Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7a9870a..2bb6a61 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -460,7 +460,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev_percent] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: move 'auto_submit_idle_job' and add account info
by Ren Wen 01 Dec '20

01 Dec '20

Move `auto_submit_idle_job` from `sched.cr` to `find_job_boot.cr`, this is a step for finding job when there's no job in conventional queues. When submitting idle job, queue is: tbox_group/idle Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 14 -------------- src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 23 +++++++++++++++++++++++ 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 26ddd92..46ded9c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -54,20 +54,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - extra_job_fields = [ - "idle_job=true", - "MASTER_FLUENTD_HOST=#{ENV["MASTER_FLUENTD_HOST"]}", - "MASTER_FLUENTD_PORT=#{ENV["MASTER_FLUENTD_PORT"]}", - ] - - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}", - extra_job_fields) if Dir.glob(full_path_patterns).size > 0 - end - def update_tbox_wtmp testbox = "" hash = Hash(String, String).new diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..5e6d0ab 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -29,3 +29,5 @@ OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) + +TEAM_ACCOUNT = "team(a)crystal.ci" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..772e82f 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -217,4 +217,27 @@ class Sched return job end + + def auto_submit_idle_job(tbox_group) + account_info = @es.get_account(TEAM_ACCOUNT).as_h + raise "Failed to verify the account: #{TEAM_ACCOUNT}" unless account_info["found"].as_bool? + my_email = account_info["my_email"] + my_name = account_info["my_name"] + my_uuid = account_info["my_uuid"] + + full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" + extra_job_fields = [ + "queue=#{tbox_group}/idle", + "my_email=#{my_email}", + "my_name=#{my_name}", + "my_uuid=#{my_uuid}", + "MASTER_FLUENTD_HOST=#{ENV["MASTER_FLUENTD_HOST"]}", + "MASTER_FLUENTD_PORT=#{ENV["MASTER_FLUENTD_PORT"]}", + ] + + Jobfile::Operate.auto_submit_job( + full_path_patterns, + "testbox: #{tbox_group}", + extra_job_fields) if Dir.glob(full_path_patterns).size > 0 + end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: add 如何登陆测试机.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in test machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...06\346\265\213\350\257\225\346\234\272.md" | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" new file mode 100644 index 0000000..774488d --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" @@ -0,0 +1,80 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1 前提条件 +请先学习: +apply-account.md，配置个人邮箱： +参考文档：[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md) +如何申请测试机，并在本地生成RSA公私钥对： +参考文档：[如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如何申请测试机.md) +submit 命令详解： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit命令详解.md) + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + suite: host-info + category: functional + + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: + +## 2.2 提交任务的方式有2种，适用于不同的登陆需求： +### 第一种：使用submit -m -c的方式提交任务： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + {"time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# + +### 第二种：使用submit的方式提交任务： + 这种方式提交的任务会在测试执行完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件如下： + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + + Regards + Compass-Ci + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + + root@vm-2p8g ~# -- 2.23.0

2 2

[PATCH v3 compass-ci] doc/manual: add how-to-log-in-the-machine.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in the machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 89 +++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/manual/how-to-log-in-the-machine.md diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md new file mode 100644 index 0000000..969257a --- /dev/null +++ b/doc/manual/how-to-log-in-the-machine.md @@ -0,0 +1,89 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1. 前提条件 +请先学习: +apply-account.md, 配置个人邮箱： +参考文档：[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md) +如何申请测试机，并在本地生成RSA公私钥对： +参考文档：[如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如何申请测试机.md) +submit 命令详解： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit命令详解.md) + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + + ```shell + suite: host-info + category: functional + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: + ``` + +## 2.2 提交任务的方式有2种，适用于不同的登陆需求： +### 第一种：使用submit -m -c的方式提交任务： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + + ```shell + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# + ``` + +### 第二种：使用submit的方式提交任务： + 这种方式提交的任务会在测试执行完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件内容如下： + + ```shell + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + Regards + Compass-Ci + ``` + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + + ```shell + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + root@vm-2p8g ~# + ``` -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] container: build script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assistant/build diff --git a/container/assistant/build b/container/assistant/build new file mode 100755 index 0000000..7778c9f --- /dev/null +++ b/container/assistant/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t debian:assistant . -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: fix submit命令详解.md
by Wu Zhende 01 Dec '20

01 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- ...submit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 71d9dfd..0edb045 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" @@ -36,7 +36,7 @@ submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbo submit iperf.yaml testbox=vm-2p8g ``` -testbox 字段的值指定需要的测试机，可以使用 `ls` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下图所示： +testbox 字段的值指定需要的测试机，可以使用 `ll` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下所示： ```shell hi8109@account-vm ~/lkp-tests/hosts% ll @@ -83,7 +83,7 @@ total 120K ### 高级用法 -submit 命令的选项如下图所示： +submit 命令的选项如下所示： ```shell hi8109@account-vm ~% submit -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] kernel_params.cr: refactor if block to one line
by Xu Xijian 01 Dec '20

01 Dec '20

In crystal language, if the last line of a function has no value, this function will return nil, so the function kernel_console can be refactored into one line. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index a558735..2cc8172 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -26,11 +26,7 @@ class Job end private def kernel_console - if os_arch == "x86_64" - return "console=ttyS0,115200 console=tty0" - else - return "" - end + return "console=ttyS0,115200 console=tty0" if os_arch == "x86_64" end private def set_kernel_params -- 2.23.0

1 1

[PATCH v2 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 01 Dec '20

01 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 62 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..dee0fdf --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,62 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# get error messages hash +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def extract_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + when /^## ______________#{@seqno}:enderr$/ + update_error_messages + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def update_error_messages + @in_stderr = false + @seqno = '' + return unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 01 Dec '20

01 Dec '20

use Math.sqrt(memory) to set container occupied memory according to system memory size, instead of setting fixed size, more reasonable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..ef3ab96 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + + +def set_available_memory + memory = `awk '($1 == "MemTotal:"){print $2/1048576}' /proc/meminfo`.to_i + + # set container available memory size, minimum size is 1g, maximum size is 30g, take the middle value according to the system memory size. + [1, 30, Math.sqrt(memory)].sort[1].to_i +end -- 2.23.0

2 2

[PATCH compass-ci] user-client/maintain: fix variable error
by Liu Yinsi 01 Dec '20

01 Dec '20

result_dict -> result_array Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index c4ffbd4..ed97999 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -70,7 +70,7 @@ def monitor_state(job_id) %w[boot extract_finished].each do |state| result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result - time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + time_list.push(Time.parse(result_array[0]['time'])) unless result_array.empty? end time_list end -- 2.23.0

1 0