- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] result-webdav/nginx.conf: expand client_max_body_size
by Lin Jiaxin 07 Dec '20

07 Dec '20

error: curl:(22) The requested URL returned error: 413 Request Entity Too Large Our biggest cgz file is 656M now, so i set client_max_body_size to 700m. /srv/initrd/pkg% find ./ -size +100M |xargs ls -alhS -rw-r--r--. 1 lkp lkp 656M Nov 4 17:52 ./nfs/openeuler/aarch64/20.03/ycsb/0.15.0-0.cgz /srv/initrd/build-pkg% find ./ -size +100M |xargs ls -alhS -rw-rw-rw-. 1 lkp lkp 383M Aug 29 04:36 ./quake2world-data-git/HEAD.cgz Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 98382bd..1ccaa92 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -12,7 +12,7 @@ http { listen 3080; server_name "result-webdav"; server_tokens off; - client_max_body_size 100m; + client_max_body_size 700m; access_log /tmp/access.log; error_log /tmp/error.log; -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/compare_matrixes.rb: fix spell error
by Lu Kaiyi 07 Dec '20

07 Dec '20

[errors] /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:482:in `block in assign_metric_change': undefined method `get_dimensions_combination' for main:Object (NoMethodError) Did you mean? get_dimentions_combination from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:476:in `each' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:476:in `assign_metric_change' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:461:in `get_metric_values' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:448:in `block in compare_metrics_values' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:447:in `each' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:447:in `compare_metrics_values' from /home/lukaiyi/compass-ci/lib/compare.rb:102:in `compare_by_template' from /home/lukaiyi/compass-ci/sbin/compare:70:in `<main>' Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 98d396f..5293989 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -495,7 +495,7 @@ end # ['openeuler 20.03', 'centos 7.6'], # ['debian sid', 'centos 7.6'] # ] -def get_dimentions_combination(dimension_list) +def get_dimensions_combination(dimension_list) dims = [] dimension_list_size = dimension_list.size (1..dimension_list_size - 1).each do |i| -- 2.23.0

1 0

[PATCH v3 compass-ci] user-client/maintain: use multi args in submit command
by Liu Yinsi 07 Dec '20

07 Dec '20

multi args changed from os os_arch os_version to os|os_arch|os_version, and if use os|os_arch|os_version, job yaml must as the first parameter. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ed97999..9f75516 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -101,8 +101,8 @@ def run_qemu end end -def submit_job(os_str, os_mount) - `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +def submit_job(os_args, os_mount) + `submit #{TEST_YAML} --my-queue 'os|os_arch|os_version=#{os_args}' testbox=#{TESTBOX} os_mount=#{os_mount}` end def traversal_os_mount @@ -113,12 +113,13 @@ end def traversal_test_os(os_mount) TEST_OS.each do |os_str| - test_rootfs(os_str, os_mount) + os_args = os_str.gsub(' ', '|') + test_rootfs(os_args, os_mount) end end -def test_rootfs(os_str, os_mount) - message = submit_job(os_str, os_mount).chomp +def test_rootfs(os_args, os_mount) + message = submit_job(os_args, os_mount).chomp job_id = message.split('=')[1] run_qemu @@ -128,7 +129,7 @@ def test_rootfs(os_str, os_mount) find_by_id(job_id) end - report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + report_all = job_id.split(', ') + os_args.split + os_mount.split + job_result write_report(report_all) end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add registry-mirrors
by Liu Yinsi 05 Dec '20

05 Dec '20

speed up docker pull speed and time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index b381d73..57ab701 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -39,6 +39,10 @@ cat > /etc/docker/daemon.json <<EOF "fluentd-async-connect": "true", "tag": "{{.Name}}" }, + "registry-mirrors": [ + "http://hub-mirror.c.163.com", + "http://docker.mirrors.ustc.edu.cn" + ], "insecure-registries": ["$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT"] } EOF -- 2.23.0

1 0

[PATCH compass-ci] scheduler: cluster: add return values
by Ren Wen 05 Dec '20

05 Dec '20

Return all ips of cluster nodes, not direct ips only. Only for one role now. Example: direct_server_ip=<direct ip> server=<public ip> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 79a13cc..1782797 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -58,7 +58,9 @@ class Sched cluster_state.each_value do |host_state| roles = host_state["roles"] direct_ips = host_state["direct_ips"] - roles_ip << "#{roles}=#{direct_ips}" + node_ip = host_state["ip"] + roles_ip << "direct_#{roles}_ips=#{direct_ips}" + roles_ip << "#{roles}=#{node_ip}" end return roles_ip.join('\n') -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare_template: friendly show compare template result in command line
by Lu Kaiyi 05 Dec '20

05 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with third-party module in python3. [usage] compare_template <compare_template.yaml> [example] compare_template compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | openeuler 20.03 standard_deviation | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- sbin/compare_template | 53 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 sbin/compare_template diff --git a/sbin/compare_template b/sbin/compare_template new file mode 100755 index 0000000..6b702c0 --- /dev/null +++ b/sbin/compare_template @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +import os +import sys +import json +import prettytable as pt + +# receive compare_template.yaml and auto pretty show compare results +class TableShow: + def __init__(self, result_dict): + self.title = result_dict['title'] + self.tables = result_dict['tables'] + self.tb = None + + def show_table(self): + for (table_title, table) in self.tables.items(): + self.tb = pt.PrettyTable() + self.set_field_names(table, table_title) + self.set_align(table_title) + self.add_row(table) + self.print_table() + + def set_field_names(self, table, table_title): + field_names = [table_title] + field_names.extend(table['average']['source'][0]) + self.tb.field_names = field_names + + def set_align(self, table_title): + for field_name in self.tb.field_names: + self.tb.align[field_name] = 'r' + self.tb.align[table_title] = 'l' + + def add_row(self, table): + row_names = ['average', 'standard_deviation'] + for row_name in row_names: + row = table[row_name]['source'][1] + row_title = ' '.join([row[0], row_name]) + format_data_row = ["%.2f" % data for data in row[1:]] + self.tb.add_row([row_title, *format_data_row]) + + def print_table(self): + print(self.title) + print(self.tb) + print() + + +if __name__ == '__main__': + template_yaml = sys.argv[1] + result_dict = json.loads(os.popen(f"compare -t {template_yaml}").read()) + table_show = TableShow(result_dict) + table_show.show_table() -- 2.23.0

1 0

[PATCH lkp-tests] stats/sysbench-threads-git: refactor regex match
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/sysbench-threads-git | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git index afe5d664..440eb798 100755 --- a/stats/sysbench-threads-git +++ b/stats/sysbench-threads-git @@ -8,7 +8,7 @@ # min: 0.04ms $stdin.each_line do |line| - next unless line =~ /^\s+total time:\s+([0-9.]+)/ + next unless line =~ /^\s+total\stime:\s+(\S+)/ - puts "total time: #{$1.to_f}s" + puts "total time: #{$1}" end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete matrix args
by Wei Jihui 04 Dec '20

04 Dec '20

if user add matrix args by overrides args, like: submit xxx.yaml 'matrix_key=matrix_value' ... it should delete matrix args from overrides, or it will keep matrix args in job.yaml all the time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2869dcb3..32329e10 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -302,6 +302,7 @@ class Job hash_job[key_array[i]] = v_array[i] end hash_job.delete(key) + @overrides.delete(key) unless @overrides[key].nil? jobs_array << hash_job end } -- 2.23.0

2 1

[PATCH lkp-tests] lib: delete the space on the left and right sides of the vertical bar
by Bai Jing 04 Dec '20

04 Dec '20

[why] The multi-args is like this "test|nr_threads", if user add it like this "test |nr_threads" or the value like "- shell1 | 1 ". So we need to delete the space Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2869dcb3..47fd56d3 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -290,11 +290,11 @@ class Job hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ - key_array = key.split('|') + key_array = key.split('|').map!{ |ele| ele.strip } [value].flatten.each do |v| next unless v =~ /^\w.*\|.*\w$/ - v_array = v.split('|') + v_array = v.split('|').map!{ |v_ele| v_ele.strip } next unless key_array.size == v_array.size hash_job = hash.clone -- 2.23.0

2 1

[PATCH compass-ci 3/3] scheduler/Sched: use JSONLogger in Sched class
by Wu Zhende 04 Dec '20

04 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 +-- src/scheduler/find_job_boot.cr | 6 +++--- src/scheduler/submit_job.cr | 2 +- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 22e0e4e..abb8a33 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -34,8 +34,7 @@ class Sched @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env - @log = JSONLogger.new - @log.set_env(env) + @log = env.get("log").as(JSONLogger) end def normalize_mac(mac : String) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6f51640..3e33604 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -192,15 +192,15 @@ class Sched begin job = @es.get_job(job_id.to_s) rescue ex - puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" - puts ex.inspect_with_backtrace + @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") + @log.warn(ex.inspect_with_backtrace) end end if job job.update({"testbox" => testbox}) job.set_result_root - puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) + @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"})) @redis.set_job(job) end return job diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index a9eb3a7..f93ad2b 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -19,7 +19,7 @@ class Sched return submit_cluster_job(job, cluster_config) end rescue ex - puts ex.inspect_with_backtrace + @log.warn(ex.inspect_with_backtrace) return [{ "job_id" => "0", "message" => ex.to_s, -- 2.23.0

1 0

[PATCH compass-ci 1/3] lib/json_logger.cr: support initialize env
by Wu Zhende 04 Dec '20

04 Dec '20

Can use JSONLogger.new(env: env) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 2b5ef6a..d136bdd 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -6,8 +6,7 @@ require "json" require "any_merge" class JSONLogger < Logger - def initialize(logdev = STDOUT, formatter = my_formatter) - @env = nil + def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) @env_info = Hash(String, String | Int32).new super(logdev, formatter: formatter) end -- 2.23.0

1 0

[PATCH compass-ci] answerback-email: add check account exists
by Luan Shengde 04 Dec '20

04 Dec '20

add check account exists for assigning/updating account avoid re-assign accounts for single email address [why] the tool may assign a new account for the email even if the email already has one. It will cause a big waste of account resource for wrong operation or test. [how] add check account exists for the email before assigning a new account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 38 ++++++++++++++------ 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 53c321d..836ad86 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -61,21 +61,19 @@ conf_info = { 'is_update_account' => false } -def init_info(mail_content, email_info, my_info) +def init_info(mail_content, email_info, my_info, my_info_es) my_info['my_email'] = mail_content.from[0] + + read_my_login_name(my_info['my_email'], my_info_es) email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - return if mail_content.attachments.empty? - email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip || nil end def read_my_login_name(my_email, my_info_es) - my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) - my_account_info = YAML.safe_load my_account_info_str - message = "No such email found from the ES: #{my_email}" - raise message unless my_account_info['found'] + my_account_info = ESQuery.new(index: 'accounts').query_by_id(my_email) - my_info_es.update my_account_info['_source'] + my_info_es.update my_account_info || {} end options = OptionParser.new do |opts| @@ -102,6 +100,7 @@ options = OptionParser.new do |opts| end my_info['my_email'] = email_address + read_my_login_name(email_address, my_info_es) end opts.on('-n name', '--name name', 'appoint name') do |name| @@ -134,7 +133,7 @@ options = OptionParser.new do |opts| return false end mail_content = Mail.read(email_file) - init_info(mail_content, email_info, my_info) + init_info(mail_content, email_info, my_info, my_info_es) end opts.on('-g', '--gen-sshkey', 'generate jumper rsa public/private key and return pubkey') do @@ -142,7 +141,6 @@ options = OptionParser.new do |opts| end opts.on('-u', '--update', 'updata configurations') do - read_my_login_name(my_info['my_email'], my_info_es) conf_info['is_update_account'] = true end @@ -223,7 +221,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == 'crystal.ci' message = 'please run the tool on z9 server' puts message @@ -240,9 +238,27 @@ def check_my_name_exist(my_info) return false end +def check_email_assigned_account(conf_info, my_info_es) + if conf_info['is_update_account'] + return true unless my_info_es.empty? + + message = "The email has not assigned an account yet.\n" + message += "Delete the '-u' option to assign a new account." + else + return true if my_info_es.empty? + + message = "The email has already assigned an account.\n" + message += "You can use option '-u' to update the account." + end + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_server return unless check_my_email(my_info) + return unless check_email_assigned_account(conf_info, my_info_es) my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v4 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assign accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 34 +++++++++++++++++------ 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..99da510 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,30 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) end def store_account_info -- 2.23.0

1 0

[PATCH v3 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assigned accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..0b94ca5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,32 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) + # my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + # YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0

1 1

[PATCH v2 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid multi accounts assignment for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when get the request for applying account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..d499840 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,6 +12,7 @@ require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +ES_PORT = ENV['ES_PORT'] || 9200 # assign uuid/account for user # when mail-robot listened new email, and the email's subject @@ -51,6 +52,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +64,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +84,31 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es['found'] + apply_info.update my_account_es['_source'] + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es['found'] + end + + def check_account_es + my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

[why] original iozone tool can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra whitespace after 2892445): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.*\d+$/ data = line.split data.each_index do |i| -- 2.23.0

1 0

[PATCH lkp-tests] Automatic compilation and configure the variables
by Hu Xuejiao 04 Dec '20

04 Dec '20

[why] Source shell_file based on different shells. [how] First, make and make install is written to the make-env. Second, invoking shell_profile(env.sh) that will distinguish different shell_files. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- make-env | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 make-env diff --git a/make-env b/make-env new file mode 100644 index 00000000..cab39288 --- /dev/null +++ b/make-env @@ -0,0 +1,6 @@ +#!/bin/bash +. lib/env.sh +make +make install + +source $(shell_profile) -- 2.23.0

3 2

[PATCH compass-ci] git_mirror.rb: disable some unnecessary messages
by Li Yuanchao 04 Dec '20

04 Dec '20

Disable some unnecessary messages, so important logs will not be covered. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index edee331..9f254f5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -235,7 +235,7 @@ class MirrorMain return if @git_info[git_repo]['is_submodule'] mirror_dir = "/srv/git/#{git_repo}.git" - show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) + show_ref_out = %x(git -C #{mirror_dir} show-ref --heads 2>/dev/null) cur_refs = { heads: {} } show_ref_out.each_line do |line| next if line.start_with? '#' -- 2.23.0

1 0

[PATCH compass-ci] auto_submit.yaml: change testbox of deploy-cci
by Li Yuanchao 04 Dec '20

04 Dec '20

'vm-2p8g' is not enough for self test. It's suggested physical machine is better. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index e0f52ba..b53b647 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,8 +1,8 @@ a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: -- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.*\d+$/ data = line.split data.each_index do |i| -- 2.23.0

2 1

[PATCH compass-ci] sparrow: add mail-robot work dir, ENV, permission
by Liu Yinsi 04 Dec '20

04 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 3 +++ sparrow/3-code/dev-env | 1 + 4 files changed, 7 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 74905f1..b381d73 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -5,6 +5,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp +grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index f5c1646..d1367cf 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -3,6 +3,7 @@ chown lkp:lkp /srv/result chown lkp:lkp /srv/initrd chown -R lkp:lkp /srv/es +chown -R mailer:team /srv/cci/Maildir chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos @@ -15,6 +16,7 @@ chmod 775 /srv/cache/netdata_cache chmod 775 /srv/cache/netdata_lib chmod 775 /srv/cci/serial/logs chmod 775 /srv/cci/serial/fluentd-pos +chmod -R 775 /srv/cci/Maildir chgrp team /srv/dc chmod g+ws /srv/dc diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index 21c2720..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -20,6 +20,9 @@ dirs=( /tftpboot /c /etc/qemu + /srv/cci/Maildir/.compass-ci/new + /srv/cci/Maildir/.compass-ci/cur + /srv/cci/Maildir/.compass-ci/sent ) mkdir -p "${dirs[@]}" diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 34cbe17..fb16000 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -38,6 +38,7 @@ LOGGING_ES_PORT: $LOGGING_ES_PORT SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +MAILDIR: /srv/cci/Maildir/.compass-ci EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v5 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 04 Dec '20

04 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- .../job_on_fail-3000.yaml => job-3000.yaml} | 0 .../{job_on_fail.yaml => job.yaml} | 1 + ...68000000-1-65535-set_get-never-never.yaml} | 0 .../redis.yaml => link_jobs1/job.yaml} | 0 .../job-1000-2-100000ss-add.yaml} | 0 .../job-10000-2-100000ss-add.yaml} | 0 .../job-20-2-6000ss-cache.yaml} | 0 .../ftq.yaml => link_jobs2/job.yaml} | 0 ...e.yaml => job-1-300-whetstone-double.yaml} | 0 ...-300-shell1.yaml => job-3-300-shell1.yaml} | 0 .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ..._files-defaults.yaml => job-defaults.yaml} | 0 spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - ...-add.yaml => job-1000-2-100000ss-add.yaml} | 0 ...add.yaml => job-10000-2-100000ss-add.yaml} | 0 ...-cache.yaml => job-20-2-6000ss-cache.yaml} | 0 .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 30 +++++-------------- 19 files changed, 9 insertions(+), 24 deletions(-) rename spec/submit/job_on_fail/{job_on_fail/job_on_fail-3000.yaml => job-3000.yaml} (100%) rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (100%) rename spec/submit/{link_jobs/redis.yaml => link_jobs1/job.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-1000-2-100000ss-add.yaml => link_jobs2/job-1000-2-100000ss-add.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-10000-2-100000ss-add.yaml => link_jobs2/job-10000-2-100000ss-add.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-20-2-6000ss-cache.yaml => link_jobs2/job-20-2-6000ss-cache.yaml} (100%) rename spec/submit/{link_jobs/ftq.yaml => link_jobs2/job.yaml} (100%) rename spec/submit/matrix/{unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml => job-1-300-whetstone-double.yaml} (100%) rename spec/submit/matrix/{unixbench-spec/unixbench-spec-3-300-shell1.yaml => job-3-300-shell1.yaml} (100%) rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) rename spec/submit/merge_yaml/{merge_included_files/merge_included_files-defaults.yaml => job-defaults.yaml} (100%) create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml rename spec/submit/separate_yaml/{ftq/ftq-1000-2-100000ss-add.yaml => job-1000-2-100000ss-add.yaml} (100%) rename spec/submit/separate_yaml/{ftq/ftq-10000-2-100000ss-add.yaml => job-10000-2-100000ss-add.yaml} (100%) rename spec/submit/separate_yaml/{ftq/ftq-20-2-6000ss-cache.yaml => job-20-2-6000ss-cache.yaml} (100%) rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml similarity index 100% rename from spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml rename to spec/submit/job_on_fail/job-3000.yaml diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs1/job.yaml similarity index 100% rename from spec/submit/link_jobs/redis.yaml rename to spec/submit/link_jobs1/job.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs2/job.yaml similarity index 100% rename from spec/submit/link_jobs/ftq.yaml rename to spec/submit/link_jobs2/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml rename to spec/submit/matrix/job-1-300-whetstone-double.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml rename to spec/submit/matrix/job-3-300-shell1.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml similarity index 100% rename from spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml rename to spec/submit/merge_yaml/job-defaults.yaml diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..c8cdcc15 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -16,20 +16,18 @@ end def traverse_file(output_dir) Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |output_dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", '-o', output_dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{output_dir}/job.yaml" ] system(*submit_cmd) traverse_file(output_dir) @@ -38,22 +36,8 @@ end describe 'submit job spec' do it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') + submit_job() end end + +submit_job -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail_bisect_result.rb: parse and send error_messages
by Cao Xueliang 04 Dec '20

04 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 2799ce2..5a208c4 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -7,7 +7,7 @@ require 'json' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @error_id = bisect_info['error_id'] + @error_messages = bisect_info['error_messages'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] @git_commit = GitCommit.new(@repo, @commit_id) @@ -32,7 +32,8 @@ class MailBisectResult After submitting a job we noticed an error response due to the commit: commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") - error_id: #{@error_id} + error_messages: + #{@error_messages} https://gitee.com/openeuler/compass-ci BODY -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/git_bisect.rb: replace error_id with error_messages
by Cao Xueliang 04 Dec '20

04 Dec '20

The error_messages will display the original error messages. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 9033db2..e2c5636 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -4,6 +4,7 @@ require 'json' require_relative 'es_query' +require_relative 'error_messages' require_relative 'sched_client' require_relative '../src/delimiter/utils' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -19,6 +20,7 @@ class GitBisect # set object property set_ids set_bad_job + set_build_pkg_dir set_upstream set_work_dir set_good_commit @@ -38,6 +40,10 @@ class GitBisect @bad_job = Utils.init_job_content(@bad_job_id) end + def set_build_pkg_dir + @build_pkg_dir = File.join('/srv', @bad_job['result_root'], 'build-pkg') + end + def set_upstream @upstream_repo = @bad_job['upstream_repo'] @upstream_commit = @bad_job['upstream_commit'] @@ -80,9 +86,10 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) + error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'job_id' => @bad_job_id, 'error_id' => @error_id] + 'job_id' => @bad_job_id, 'error_messages' => error_messages.join("\n")] end # first search the good commit in db -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.*\d+$/ data = line.split data.each_index do |i| -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] stats/crystal: sysbench-threads-git to sysbench-threads-git.cr
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/sysbench-threads-git.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 stats/crystal/sysbench-threads-git.cr diff --git a/stats/crystal/sysbench-threads-git.cr b/stats/crystal/sysbench-threads-git.cr new file mode 100644 index 00000000..91707e87 --- /dev/null +++ b/stats/crystal/sysbench-threads-git.cr @@ -0,0 +1,14 @@ +#!/usr/bin/env crystal + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +while (line = STDIN.gets) + next unless line =~ /^\s+total\stime:\s+(\S+)/ + + puts "total time: #{$1}" +end -- 2.23.0

1 0

[PATCH v2 compass-ci] How to add a machine to compass-ci
by Zhang Dewan 04 Dec '20

04 Dec '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> How a new machine add to compass-ci Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- ...00\350\210\254\350\277\207\347\250\213.md" | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 "doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" diff --git "a/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" new file mode 100644 index 0000000..340ffea --- /dev/null +++ "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" @@ -0,0 +1,54 @@ + +这个过程可以使用“注册--发现”的模型来理解。 + +## 一台机器成为测试机: +- 需要注册到compass-ci +- 需要能够执行调度器（scheduler）给它的任务 + +## 注册 +### 1. 注册信息保存在哪 +测试机需要注册到z9上面的redis-db. + +x.x.x.x:xxxx> `hgetall "sched/mac2host"` # 登录redis, 这个命令可以查看已经注册的测试机信息 + +### 2. 怎么注册 +使用scheduler服务接口注册，模板： +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}"` +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${api_queues}"` + +可以看到，这两个调用需要三个参数：mac、hostname、api_queues + +#### 2.1 接口参数说明 +##### 2.1.1 mac +当前我们的机器都是局域网内的资源，所以没有比使用mac注册更直接、方便的了。 + +###### 2.1.1.1 如何获取mac地址，这里用TaiShan服务器举例： +- 登录bmc，打开信息--系统信息--网络--网卡--第一张网卡--port1，即可查看。 +- 登录bios，进入Advanced--Network Device List，即可查看。 +- 使用一台和机器的bmc同网段的机器，执行以下命令即可查看： +`ipmitool -I lanplus -H $cur_ip -U $iBMC_user -P $iBMC_passwd raw 0x30 0x90 0x01 0x03 0x0${nic_idx} 0x00|awk '{print $5,$6,$7,$8,$9,$10}' |sed 's/\s/\-/g'` +- 给服务器安装os，登录上去，执行命令，即可查看： +`ip link | grep "^2:" -A 1| grep link |awk '{print $2}` + +##### 2.1.2 hostname +如同每一张网卡的mac是规则且唯一的，hostname作为资源的标识也是如此。 + +###### 2.1.2.1 如何定义hostname，这里还是以Taishan服务器为例： +` taishan200-2280-${cpuNum}s${coreNum}p-${memSize:-256}g--${aN}` +> taishan200-2280 服务器型号 +> cpuNum cpu个数 +> coreNum cpu核数 +> memSize 内存大小 +> aN 该类型机器编号 + +#### 2.1.3 api_queues +作为单台机器的组标识，一般取hostname的前缀。 +比如，taishan200-2280-2s64p-256g--a100, 它的api_queues为：taishan200-2280-2s64p-256g + +## 发现 +### 1. 怎么发现测试机并给它分配任务 +测试机器需要将启动方式设置pxe启动，然后在启动环节向scheduler发起请求，获取启动参数和任务信息。 + +### 2. 设置PXE启动 +- 华为服务器默认pxe启动，无需配置。 +- 其他设备，根据产品类型，按文档操作即可。 -- 2.23.0

2 1

[PATCH lkp-tests] stats/crystal: sysbench-threads-git to sysbench-threads-git.cr
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/sysbench-threads-git.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 stats/crystal/sysbench-threads-git.cr diff --git a/stats/crystal/sysbench-threads-git.cr b/stats/crystal/sysbench-threads-git.cr new file mode 100644 index 00000000..5d9dcc10 --- /dev/null +++ b/stats/crystal/sysbench-threads-git.cr @@ -0,0 +1,14 @@ +#!/usr/bin/env crystal + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +while (line = STDIN.gets) + next unless line =~ /^\s+total time:\s+([0-9.]+)/ + + puts "total time: #{$1.to_f}s" +end -- 2.23.0

2 1

[PATCH v2 compass-ci] user-client/maintain: add '|' for multi args
by Liu Yinsi 04 Dec '20

04 Dec '20

multi args changed from os os_arch os_version to os|os_arch|os_version, and if use os|os_arch|os_version, job yaml must as the first parameter. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ed97999..9f75516 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -101,8 +101,8 @@ def run_qemu end end -def submit_job(os_str, os_mount) - `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +def submit_job(os_args, os_mount) + `submit #{TEST_YAML} --my-queue 'os|os_arch|os_version=#{os_args}' testbox=#{TESTBOX} os_mount=#{os_mount}` end def traversal_os_mount @@ -113,12 +113,13 @@ end def traversal_test_os(os_mount) TEST_OS.each do |os_str| - test_rootfs(os_str, os_mount) + os_args = os_str.gsub(' ', '|') + test_rootfs(os_args, os_mount) end end -def test_rootfs(os_str, os_mount) - message = submit_job(os_str, os_mount).chomp +def test_rootfs(os_args, os_mount) + message = submit_job(os_args, os_mount).chomp job_id = message.split('=')[1] run_qemu @@ -128,7 +129,7 @@ def test_rootfs(os_str, os_mount) find_by_id(job_id) end - report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + report_all = job_id.split(', ') + os_args.split + os_mount.split + job_result write_report(report_all) end -- 2.23.0

1 0

[PATCH compass-ci] change file mode to be the same with others
by Ren Wen 04 Dec '20

04 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/json_logger.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 src/lib/json_logger.cr diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] scheduler/jobfile_operate.cr: refactor create_job_cpio
by Wu Zhende 04 Dec '20

04 Dec '20

Separate "create_job_cpio" into a new file. Use class Sched, can use Sched's instance variable. Also can simplify jobfile_operate.cr. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/create_job_cpio.cr | 140 +++++++++++++++++++++++++++++++ src/scheduler/find_job_boot.cr | 2 +- src/scheduler/jobfile_operate.cr | 132 ----------------------------- 4 files changed, 142 insertions(+), 133 deletions(-) create mode 100644 src/scheduler/create_job_cpio.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a9f893f..22e0e4e 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -20,6 +20,7 @@ require "../scheduler/find_next_job_boot" require "../scheduler/close_job" require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" +require "../scheduler/create_job_cpio" class Sched property es diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr new file mode 100644 index 0000000..5c1be7e --- /dev/null +++ b/src/scheduler/create_job_cpio.cr @@ -0,0 +1,140 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# shellwords require from '/c/lkp-tests/lib/' +require "shellwords" +require "file_utils" +require "json" +require "yaml" + +require "./jobfile_operate" + +class Sched + private def prepare_dir(file_path : String) + file_path_dir = File.dirname(file_path) + unless File.exists?(file_path_dir) + FileUtils.mkdir_p(file_path_dir) + end + end + + private def valid_shell_variable?(key) + key =~ /^[a-zA-Z_]+[a-zA-Z0-9_]*$/ + end + + private def create_job_sh(job_sh_content : Array(JSON::Any), path : String) + File.open(path, "w", File::Permissions.new(0o775)) do |file| + file.puts "#!/bin/sh\n\n" + + job_sh_content.each do |line| + if line.as_a? + line.as_a.each { |val| file.puts val } + else + file.puts line + end + end + + file.puts "\"$@\"" + end + end + + private def shell_escape(val) + val = val.join "\n" if val.is_a?(Array) + + if val.nil? || val.empty? + return nil + elsif val =~ /^[+-]?([0-9]*\.?[0-9]+|[0-9]+\.?[0-9]*)([eE][+-]?[0-9]+)?$/ + return val + elsif !val.includes?("'") && !val.includes?("$") + return "'#{val}'" + elsif !val.includes?('"') + return "\"#{val}\"" + else + return Shellwords.shellescape(val) + end + end + + private def parse_one(script_lines, key, val) + return false if val.as_h? || !valid_shell_variable?(key) + + value = shell_escape(val.as_a? || val.to_s) + script_lines << "\texport #{key}=" + value if value + end + + private def sh_export_top_env(job_content : Hash) + script_lines = ["export_top_env()", "{"] of String + + job_content.each { |key, val| parse_one(script_lines, key, val) } + + script_lines << "\n" + script_lines << "\t[ -n \"$LKP_SRC\" ] ||" + script_lines << "\texport LKP_SRC=/lkp/${user:-lkp}/src" + script_lines << "}\n\n" + + script_lines = script_lines.to_s + script_lines = JSON.parse(script_lines) + end + + def create_job_cpio(job_content : JSON::Any, base_dir : String) + job_content = job_content.as_h + + # put job2sh in an array + if job_content.has_key?("job2sh") + tmp_job_sh_content = job_content["job2sh"] + + job_sh_array = [] of JSON::Any + tmp_job_sh_content.as_h.each do |_key, val| + next if val == nil + job_sh_array += val.as_a + end + else + job_sh_array = [] of JSON::Any + end + + # generate job.yaml + temp_yaml = base_dir + "/#{job_content["id"]}/job.yaml" + prepare_dir(temp_yaml) + + # no change to <object> content { "#! jobs/pixz.yaml": null } + # - this will create a <'#! jobs/pixz.yaml':> in the yaml file + # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml + # tested : no effect to job.sh + File.open(temp_yaml, "w") do |file| + YAML.dump(job_content, file) + end + + # generate unbroken job shell content + sh_export_top_env = sh_export_top_env(job_content) + job_sh_content = sh_export_top_env.as_a + job_sh_array + + # generate job.sh + job_sh = base_dir + "/#{job_content["id"]}/job.sh" + create_job_sh(job_sh_content.to_a, job_sh) + + job_dir = base_dir + "/#{job_content["id"]}" + + if job_sh_array.empty? + lkp_src = Jobfile::Operate.prepare_lkp_tests(job_content["lkp_initrd_user"], + job_content["os_arch"]) + + cmd = "#{lkp_src}/sbin/create-job-cpio.sh #{temp_yaml}" + idd = `#{cmd}` + else + cmd = "./create-job-cpio.sh #{job_dir}" + idd = `#{cmd}` + end + + # if the create job cpio failed, what to do? + puts idd if idd.match(/ERROR/) + + # create result dir and copy job.sh, job.yaml and job.cgz to result dir + src_dir = File.dirname(temp_yaml) + dst_dir = File.join("/srv", job_content["result_root"].to_s) + FileUtils.mkdir_p(dst_dir) + + # the job.yaml is not final version + files = ["#{src_dir}/job.sh", + "#{src_dir}/job.yaml", + "#{src_dir}/job.cgz"] + FileUtils.cp(files, dst_dir) + end +end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ad48f27..6f51640 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -75,7 +75,7 @@ class Sched job = get_job_from_queues(queues, host) if job - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) end return boot_content(job, boot_type) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..df79d3b 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -5,139 +5,7 @@ require "file_utils" require "json" require "yaml" -# require from '/c/lkp-tests/lib/' -require "shellwords" - -LKP_PATH = "/c/lkp-tests" -if ENV["LKP_SRC"] != LKP_PATH - raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} #{LKP_PATH}" -end - module Jobfile::Operate - def self.prepare_dir(file_path : String) - file_path_dir = File.dirname(file_path) - unless File.exists?(file_path_dir) - FileUtils.mkdir_p(file_path_dir) - end - end - - def self.valid_shell_variable?(key) - key =~ /^[a-zA-Z_]+[a-zA-Z0-9_]*$/ - end - - def self.create_job_sh(job_sh_content : Array(JSON::Any), path : String) - File.open(path, "w", File::Permissions.new(0o775)) do |file| - file.puts "#!/bin/sh\n\n" - job_sh_content.each do |line| - if line.as_a? - line.as_a.each { |val| file.puts val } - else - file.puts line - end - end - file.puts "\"$@\"" - end - end - - def self.shell_escape(val) - val = val.join "\n" if val.is_a?(Array) - - if val.nil? || val.empty? - return nil - elsif val =~ /^[+-]?([0-9]*\.?[0-9]+|[0-9]+\.?[0-9]*)([eE][+-]?[0-9]+)?$/ - return val - elsif !val.includes?("'") && !val.includes?("$") - return "'#{val}'" - elsif !val.includes?('"') - return "\"#{val}\"" - else - return Shellwords.shellescape(val) - end - end - - def self.parse_one(script_lines, key, val) - return false if val.as_h? || !valid_shell_variable?(key) - - value = shell_escape(val.as_a? || val.to_s) - script_lines << "\texport #{key}=" + value if value - end - - def self.sh_export_top_env(job_content : Hash) - script_lines = ["export_top_env()", "{"] of String - - job_content.each { |key, val| parse_one(script_lines, key, val) } - - script_lines << "\n" - script_lines << "\t[ -n \"$LKP_SRC\" ] ||" - script_lines << "\texport LKP_SRC=/lkp/${user:-lkp}/src" - script_lines << "}\n\n" - - script_lines = script_lines.to_s - script_lines = JSON.parse(script_lines) - end - - def self.create_job_cpio(job_content : JSON::Any, base_dir : String) - job_content = job_content.as_h - - # put job2sh in an array - if job_content.has_key?("job2sh") - tmp_job_sh_content = job_content["job2sh"] - - job_sh_array = [] of JSON::Any - tmp_job_sh_content.as_h.each do |_key, val| - next if val == nil - job_sh_array += val.as_a - end - else - job_sh_array = [] of JSON::Any - end - - # generate job.yaml - temp_yaml = base_dir + "/#{job_content["id"]}/job.yaml" - prepare_dir(temp_yaml) - - # no change to <object> content { "#! jobs/pixz.yaml": null } - # - this will create a <'#! jobs/pixz.yaml':> in the yaml file - # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml - # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) - end - - # generate unbroken job shell content - sh_export_top_env = sh_export_top_env(job_content) - job_sh_content = sh_export_top_env.as_a + job_sh_array - - # generate job.sh - job_sh = base_dir + "/#{job_content["id"]}/job.sh" - create_job_sh(job_sh_content.to_a, job_sh) - - job_dir = base_dir + "/#{job_content["id"]}" - - if job_sh_array.empty? - lkp_src = prepare_lkp_tests(job_content["lkp_initrd_user"], - job_content["os_arch"]) - - cmd = "#{lkp_src}/sbin/create-job-cpio.sh #{temp_yaml}" - idd = `#{cmd}` - else - cmd = "./create-job-cpio.sh #{job_dir}" - idd = `#{cmd}` - end - - # if the create job cpio failed, what to do? - puts idd if idd.match(/ERROR/) - # create result dir and copy job.sh, job.yaml and job.cgz to result dir - src_dir = File.dirname(temp_yaml) - dst_dir = File.join("/srv", job_content["result_root"].to_s) - FileUtils.mkdir_p(dst_dir) - # the job.yaml is not final version - files = ["#{src_dir}/job.sh", - "#{src_dir}/job.yaml", - "#{src_dir}/job.cgz"] - FileUtils.cp(files, dst_dir) - end - def self.unzip_cgz(source_path : String, target_path : String) FileUtils.mkdir_p(target_path) cmd = "cd #{target_path};gzip -dc #{source_path}|cpio -id" -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/common.rb: add build_pkg_error_id function
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/common.rb | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/common.rb b/lib/common.rb index 91997baa..cc64eb17 100755 --- a/lib/common.rb +++ b/lib/common.rb @@ -319,3 +319,25 @@ end def delete_file_if_exist(file) File.delete(file) if File.exist?(file) end + +def build_pkg_error_id(line) + line = line.chomp + line = line.gsub(/\b[3-9]\.[0-9]+[-a-z0-9.]+/, "#") # linux version: 3.17.0-next-20141008-g099669ed + line = line.gsub(/\b[1-9][0-9]-[A-Z][a-z]+-[0-9]{4}\b/, "#") # Date: 28-Dec-2013 + line = line.gsub(/\b0x[0-9a-f]+\b/, "#") # hex number + line = line.gsub(/\b[a-f0-9]{40}\b/, "#") # SHA-1 + line = line.gsub(/\b[0-9][0-9.]*/, "#") # number + line = line.gsub(/#x\b/, "0x") + line = line.gsub(/[\\"$]/, "~") + line = line.gsub(/[ \t]/, " ") + line = line.gsub(/\ \ +/, " ") + line = line.gsub(/([^a-zA-Z0-9])\ /, "\\1") + line = line.gsub(/\ ([^a-zA-Z])/, "\\1") + line = line.gsub(/^\ /, "") + line = line.gsub(/^-/, "") + line = line.gsub(/\ _/, "_") + line = line.tr(" ", "-") + line = line.gsub(/[-_.,;:#!\[(]+$/, "") + line = line.gsub(/([-_.,;:#!]){3,}/, ":") + line +end -- 2.23.0

1 0

[PATCH lkp-tests 1/2] stats/build-pkg: rename common_error_id function and move it to ../lib/common
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 28 +++++----------------------- 1 file changed, 5 insertions(+), 23 deletions(-) diff --git a/stats/build-pkg b/stats/build-pkg index 9eef8c91..90f27656 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -1,27 +1,9 @@ #!/usr/bin/env ruby -require 'set' -def common_error_id(line) - line = line.chomp - line = line.gsub(/\b[3-9]\.[0-9]+[-a-z0-9.]+/, "#") # linux version: 3.17.0-next-20141008-g099669ed - line = line.gsub(/\b[1-9][0-9]-[A-Z][a-z]+-[0-9]{4}\b/, "#") # Date: 28-Dec-2013 - line = line.gsub(/\b0x[0-9a-f]+\b/, "#") # hex number - line = line.gsub(/\b[a-f0-9]{40}\b/, "#") # SHA-1 - line = line.gsub(/\b[0-9][0-9.]*/, "#") # number - line = line.gsub(/#x\b/, "0x") - line = line.gsub(/[\\"$]/, "~") - line = line.gsub(/[ \t]/, " ") - line = line.gsub(/\ \ +/, " ") - line = line.gsub(/([^a-zA-Z0-9])\ /, "\\1") - line = line.gsub(/\ ([^a-zA-Z])/, "\\1") - line = line.gsub(/^\ /, "") - line = line.gsub(/^-/, "") - line = line.gsub(/\ _/, "_") - line = line.tr(" ", "-") - line = line.gsub(/[-_.,;:#!\[(]+$/, "") - line = line.gsub(/([-_.,;:#!]){3,}/, ":") - line -end +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require 'set' +require "#{LKP_SRC}/lib/common" error_ids = Set.new in_stderr = false @@ -40,7 +22,7 @@ while (line = STDIN.gets) next unless in_stderr next unless line.downcase =~ /error|warning/ - error_ids << common_error_id(line) + error_ids << build_pkg_error_id(line) end error_ids.each do |id| -- 2.23.0

1 0

[PATCH compass-ci] lib/error_messages.rb: add obtain_error_messages_by_errorid function
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 7bf3aa8..3a74d29 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -3,10 +3,13 @@ # frozen_string_literal: true require 'set' +require_relative "#{ENV['LKP_SRC']}/lib/common" # usage: # foo = ErrorMessages.new(build-pkg) # build-pkg is a file that includes errors generated by make +# errorid_array = foo.obtain_error_messages_by_errorid(errorid) +# errorid_array includes all error_messages that error same as errorid # error_messages_hash = foo.obtain_error_messages # error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } # output: @@ -37,6 +40,17 @@ class ErrorMessages return @error_messages end + def obtain_error_messages_by_errorid(errorid) + error_messages_by_errorid = [] + error_messages = obtain_error_messages + error_messages.each do | k, v | + if ("build-pkg." + build_pkg_error_id(k)) == errorid + error_messages_by_errorid << v + end + end + error_messages_by_errorid + end + private def extract_error_message(line) -- 2.23.0

1 0

[PATCH compass-ci] user-client/maintain: add '|' for multi args
by Liu Yinsi 03 Dec '20

03 Dec '20

multi args changed from os os_arch os_version to os|os_arch|os_version, and if use os|os_arch|os_version, job yaml must as the first parameter. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ed97999..9f75516 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -101,8 +101,8 @@ def run_qemu end end -def submit_job(os_str, os_mount) - `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +def submit_job(os_args, os_mount) + `submit #{TEST_YAML} --my-queue 'os|os_arch|os_version=#{os_args}' testbox=#{TESTBOX} os_mount=#{os_mount}` end def traversal_os_mount @@ -113,12 +113,13 @@ end def traversal_test_os(os_mount) TEST_OS.each do |os_str| - test_rootfs(os_str, os_mount) + os_args = os_str.gsub(' ', '|') + test_rootfs(os_args, os_mount) end end -def test_rootfs(os_str, os_mount) - message = submit_job(os_str, os_mount).chomp +def test_rootfs(os_args, os_mount) + message = submit_job(os_args, os_mount).chomp job_id = message.split('=')[1] run_qemu @@ -128,7 +129,7 @@ def test_rootfs(os_str, os_mount) find_by_id(job_id) end - report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + report_all = job_id.split(', ') + os_args.split + os_mount.split + job_result write_report(report_all) end -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 34 ++++++---------------------------- 1 file changed, 6 insertions(+), 28 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..75ad9798 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -16,44 +16,22 @@ end def traverse_file(output_dir) Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", '-o', output_dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(out_dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH compass-ci] src/scheduler: delete job2sh field in job.yaml
by Zhang Yu 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/scheduler/jobfile_operate.cr | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..82ed885 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -80,16 +80,13 @@ module Jobfile::Operate job_content = job_content.as_h # put job2sh in an array - if job_content.has_key?("job2sh") - tmp_job_sh_content = job_content["job2sh"] - - job_sh_array = [] of JSON::Any + job_sh_array = [] of JSON::Any + tmp_job_sh_content = job_content.delete("job2sh") + if tmp_job_sh_content tmp_job_sh_content.as_h.each do |_key, val| next if val == nil job_sh_array += val.as_a end - else - job_sh_array = [] of JSON::Any end # generate job.yaml -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix upload in container
by Wang Yong 03 Dec '20

03 Dec '20

docker can not use mount operation, so now use curl to upload build-pkg cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index cde2f0edd..38d8bace9 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -18,6 +18,8 @@ ## See: https://wiki.archlinux.org/index.php/Makepkg . $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/distro/common check_vars() { @@ -29,6 +31,13 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +mount_pkg_mnt() +{ + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -39,9 +48,7 @@ mount_dest() PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + is_docker || mount_pkg_mnt } get_pkgfile() @@ -139,10 +146,15 @@ build_source_pkg() replace_tar_source cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] || exit 1 + create_softlink + + is_docker || exit 0 + upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } check_vars -- 2.23.0

2 1

[PATCH v3 compass-ci] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..5dc68c8 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,28 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job job_id - +require 'optparse' require_relative '../lib/mail_job_result.rb' -job_id = ARGV[0] +# mail job results to job submitter +# require the job had submitted job.yaml include email + +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: mail-job job_id' + opts.separator 'eg: mail-job crystal.123012' + opts.separator '' + opts.separator 'options:' + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +argv = ARGV.empty? ? ['-h'] : ARGV + +opt_parser.parse!(argv) + +job_id = argv[0] mail = MailJobResult.new(job_id) mail.send_mail -- 2.23.0

1 0

[PATCH compass-ci] container/conserver: change executable script
by Cui Yili 03 Dec '20

03 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/build | 2 +- container/conserver/{conserver-head.cf => conserver.cf} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename container/conserver/{conserver-head.cf => conserver.cf} (100%) diff --git a/container/conserver/build b/container/conserver/build index 25c785c..0700c5e 100755 --- a/container/conserver/build +++ b/container/conserver/build @@ -2,6 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -sh generate_conserver || exit 1 +./generate_conserver.rb || exit 1 docker build -t conserver_server . diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver.cf similarity index 100% rename from container/conserver/conserver-head.cf rename to container/conserver/conserver.cf -- 2.23.0

1 0

[PATCH compass-ci] manual/如何申请测试机.md: change the borrow-1h.yaml to latest
by Zhang Yale 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...257\267\346\265\213\350\257\225\346\234\272.md" | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" index d70136f..85d385d 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" @@ -209,14 +209,14 @@ suite: borrow testcase: borrow + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> # sleep at the bottom sleep: 1h hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml -- 2.23.0

2 2

[PATCH compass-ci 2/2] mail-robot: update start
by Luan Shengde 03 Dec '20

03 Dec '20

add MAILDIR to default yaml file avoid change start file if there is a new maildir. add ES_PORT Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 1c55dd3..f8078b6 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -14,10 +14,11 @@ names = Set.new %w[ SEND_MAIL_PORT SCHED_HTTPS_HOST SCHED_HTTPS_PORT + ES_PORT + MAILDIR ] defaults = relevant_defaults(names) -defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' env = docker_env(defaults) -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: add check assigned account for email
by Luan Shengde 03 Dec '20

03 Dec '20

avoid multi accounts assignment for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. this is a big waste of account resource [how] when get the request for applying account, check whether the email already has an account before applying account. apply a new one if there is none return the old one if there is already one for the email. and update my_info if the email has new: my_name my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 37 +++++++++++++++++------ 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..24f41bc 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,6 +12,7 @@ require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +ES_PORT = ENV['ES_PORT'] || 9200 # assign uuid/account for user # when mail-robot listened new email, and the email's subject @@ -51,6 +52,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +64,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +84,32 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es['found'] + apply_info.update my_account_es['_source'] + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es['found'] + # return acct_info + end + + def check_account_es + my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: update submit命令详解.md
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- ...bmit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 6 ++++++ 1 file changed, 6 insertions(+) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 0edb045..66d8cc4 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" @@ -129,6 +129,12 @@ options: submit -a iperf.yaml ``` + ``` + 可添加的文件及目录：lkp-tests/*/$program ， lkp-tests/*/$program/* ， lkp-tests/*/*/$program ， lkp-tests/*/*/$program/* 。 + $program 的值是 $suite ， $suite-dev ， $suite.aarch64 或 $suite.x86_64 。 + 当 suite 为 makepkg ， makepkg-deps ， pack-deps ， cci-makepkg 或 cci-depends 时，$program = $benchmark 。 + ``` + * **-m的用法** 使用 -m 参数可以启动任务监控功能，并将任务执行过程中的各种状态信息打印到控制台上，方便用户实时监控测试任务的执行过程。 -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: unified indent
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 148 ++++++++++++++++++++++++------------------------ 1 file changed, 74 insertions(+), 74 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index cde2f0ed..b2679850 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -21,60 +21,60 @@ check_vars() { - [ -n "$os" ] || die "os is empty" - [ -n "$os_arch" ] || die "os_arch is empty" - [ -n "$os_version" ] || die "os_version is empty" - [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" - [ -n "$upstream_commit" ] || die "upstream_commit is empty" - [ -n "$os_mount" ] || die "os_mount is empty" + [ -n "$os" ] || die "os is empty" + [ -n "$os_arch" ] || die "os_arch is empty" + [ -n "$os_version" ] || die "os_version is empty" + [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" + [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { - # the same image is mounted to cifs and nfs, the generated cgz files - # are stored in the nfs directory. - [[ "$os_mount" = "cifs" ]] && os_mount="nfs" - pack_to=${os_mount}/${os}/${os_arch}/${os_version} + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} - PKG_MNT=/initrd/build-pkg - mkdir -p "$PKG_MNT" + PKG_MNT=/initrd/build-pkg + mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } } get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" -} + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" + } request_pkg() { - local pkgrepo="${pkgbuild_repo%%//*}" - local pkgpath="./" - [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" - - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') - - for pkgfile in ${filelist[*]} - do - get_pkgfile "${pkgrepo}" "$pkgfile" - done - - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" - } - - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" + local pkgrepo="${pkgbuild_repo%%//*}" + local pkgpath="./" + [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" + + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') + + for pkgfile in ${filelist[*]} + do + get_pkgfile "${pkgrepo}" "$pkgfile" + done + + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + + [ -f PKGBUILD ] || die "PKGBUILD not exist" + [ -s PKGBUILD ] || die "PKGBUILD is empty" } create_softlink() @@ -95,54 +95,54 @@ create_softlink() replace_source() { - sed -i "s|^source=[^)]*|$1|g" "$2" + sed -i "s|^source=[^)]*|$1|g" "$2" } # replace *.tar.* address from source field in PKGBUILD replace_tar_source() { - echo "source=(" > PKGBUILD.src - for url in ${source[@]} - do - echo "$url" | egrep '.+\.tar' && { - local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" - # check whether local server exists *.tar.* file - wget -q --spider "${local_url}" - [ "$?" == 0 ] || continue + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue - url="\"${local_url}\"" + url="\"${local_url}\"" } - echo "$url" >> PKGBUILD.src - done - echo ")" >> PKGBUILD.src + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src } build_source_pkg() { - local repo_dir="" - source PKGBUILD - [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { - - if [[ "$source" = *::* ]]; then - repo_dir="${source%%::*}::" - else - repo_dir="$(basename ${upstream_url})" - repo_dir="${repo_dir%.git*}::" - fi - - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" - replace_source "${upstream_source}" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" - } + local repo_dir="" + source PKGBUILD + [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { + + if [[ "$source" = *::* ]]; then + repo_dir="${source%%::*}::" + else + repo_dir="$(basename ${upstream_url})" + repo_dir="${repo_dir%.git*}::" + fi + + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" + } - replace_tar_source + replace_tar_source - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] && create_softlink } check_vars -- 2.23.0

1 0

[PATCH v5 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 49 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 54 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 54 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 54 ++++++++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 85 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------- .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------- .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------- spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 52 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 52 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 46 ++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ----- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 54 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 54 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 54 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 563 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..658b0740 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..85ed3725 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..345033e0 --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..fc926d47 --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..b8abc5b9 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,40 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +82,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..a0ff2d27 --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..8ed2467d --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..76656b1f --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,46 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4a24064c --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..691e1fb1 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..f344ae95 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/3-code: fix submit command not found
by Liu Yinsi 03 Dec '20

03 Dec '20

[why] when execute submit command in a new machine, error message: [root@localhost 3-code]# submit -bash: submit: command not found because of no '' around EOF, the command cat will parse variable, cause PATH=$PATH:/sbin:/sbin:/bin, but we want is PATH=$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin [how] use 'EOF' to avoid $CCI_SRC, $LKP_SRC to be parsed in cat command. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 444281e..34cbe17 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -40,7 +40,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS EOF -cat > /etc/profile.d/compass.sh <<EOF +cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci export REPO_SRC=/c/upstream-repos -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..5dc68c8 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,28 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job job_id - +require 'optparse' require_relative '../lib/mail_job_result.rb' -job_id = ARGV[0] +# mail job results to job submitter +# require the job had submitted job.yaml include email + +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: mail-job job_id' + opts.separator 'eg: mail-job crystal.123012' + opts.separator '' + opts.separator 'options:' + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +argv = ARGV.empty? ? ['-h'] : ARGV + +opt_parser.parse!(argv) + +job_id = argv[0] mail = MailJobResult.new(job_id) mail.send_mail -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: fix submit command not found
by Liu Yinsi 03 Dec '20

03 Dec '20

[why] when execute submit command in a new machine, error message: [root@localhost 3-code]# submit -bash: submit: command not found [how] use 'EOF' to avoid $CCI_SRC, $LKP_SRC to be parsed in cat command and cause failed to set ENV $PATH. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 444281e..34cbe17 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -40,7 +40,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS EOF -cat > /etc/profile.d/compass.sh <<EOF +cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci export REPO_SRC=/c/upstream-repos -- 2.23.0

2 2

[PATCH v3 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 03 Dec '20

03 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..17197e7 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,17 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic configurations, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... -- 2.23.0

3 3

[PATCH compass-ci] doc/manual: update submit命令详解.md
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- ...submit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 4 ++++ 1 file changed, 4 insertions(+) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 0edb045..1b6ce8a 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" @@ -129,6 +129,10 @@ options: submit -a iperf.yaml ``` + 可添加的文件及目录：lkp-tests/*/$program ， lkp-tests/*/$program/* ， lkp-tests/*/*/$program ， lkp-tests/*/*/$program/* 。 + $program 的值可以是 $suite ， $suite-dev ， $suite.aarch64 ， $suite.x86_64 。 + 当 suite 为 makepkg ， makepkg-deps ， pack-deps ， cci-makepkg 或 cci-depends 时，$program = $benchmark 。 + * **-m的用法** 使用 -m 参数可以启动任务监控功能，并将任务执行过程中的各种状态信息打印到控制台上，方便用户实时监控测试任务的执行过程。 -- 2.23.0

2 1

[PATCH v4 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 88 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 +++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 596 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..ba732182 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..4b6bbacc --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..9476bf3c --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..e1f2bbf6 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

2 2

[PATCH compass-ci 1/2] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..85e37a1 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,32 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job job_id - +require 'optparse' require_relative '../lib/mail_job_result.rb' -job_id = ARGV[0] +# mail job results to job submitter +# require the job had submitted job.yaml include email + +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: mail-job job_id' + opts.separator 'eg: mail-job crystal.123012' + opts.separator '' + opts.separator 'options:' + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +argv = if ARGV == [] + ['-h'] + else + ARGV + end + +opt_parser.parse!(argv) + +job_id = argv[0] mail = MailJobResult.new(job_id) mail.send_mail -- 2.23.0

3 5

[PATCH v2 compass-ci] lib/mail_job_result.rb: fix NoMethodError
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] /home/lwt/compass-ci/lib/mail_job_result.rb:45:in `query_job': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 29a9c1a..50e2f53 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -33,7 +33,7 @@ class MailJobResult def set_submitter_info job = query_job - exit unless job['email'] + exit unless job && job['email'] @submitter_email = job['email'] @result_root = job['result_root'] @@ -42,6 +42,11 @@ class MailJobResult def query_job es = ESQuery.new query_result = es.multi_field_query({ 'id' => @job_id }) + if query_result['hits']['hits'].empty? + warn "Non-existent job: #{@job_id}" + return nil + end + query_result['hits']['hits'][0]['_source'] end end -- 2.23.0

1 1

[PATCH compass-ci] scheduler: fix the kernel_params values displayed in job.yaml
by Bai Jing 03 Dec '20

03 Dec '20

[original values] kernel_params: ' user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro rdinit=/sbin/init prompt_ramdisk=0 initrd=20201022.0.cgz initrd=run-ipconfig_20200904.cgz initrd=modules-4.19.90-2003.cgz initrd=headers-4.19.90-2003.cgz initrd=mpstat_20201104.cgz initrd=spinlock_20201104.cgz initrd=git-1.cgz initrd=1.0-1.cgz initrd=lkp-aarch64.cgz initrd=job.cgz ' [now values] kernel_params: - user=lkp - job=/lkp/scheduled/job.yaml - RESULT_ROOT=/result/job - rootovl - ip=dhcp - ro - rdinit=/sbin/init - prompt_ramdisk=0 - initrd=20201022.0.cgz - initrd=run-ipconfig_20200904.cgz - initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz - initrd=headers-4.19.90-2003.4.0.0036.oe1.aarch64.cgz - initrd=mpstat_20201104.cgz - initrd=spinlock_20201104.cgz - initrd=git-1.cgz - initrd=1.0-1.cgz - initrd=lkp-aarch64.cgz - initrd=job.cgz Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/scheduler/find_job_boot.cr | 4 ++-- src/scheduler/kernel_params.cr | 4 +++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 3b68b59..ad48f27 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -135,8 +135,8 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri - response += job.kernel_params + response += job.kernel_uri + " " + response += Array(String).from_json(job.kernel_params).join(" ") response += "\nboot\n" return response diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 0afe692..76606aa 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -30,7 +30,9 @@ class Job end private def set_kernel_params - self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" + kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" + kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) + @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) end end -- 2.23.0

1 0

[PATCH v5 compass-ci] 优化部分文档
by Li Ping 03 Dec '20

03 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.en.md | 6 +++--- README.md => README.zh.md | 4 ++-- ...\263\350\257\267\346\265\213\350\257\225\346\234\272.md" | 2 -- ...\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 6 +++--- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 5 files changed, 9 insertions(+), 11 deletions(-) rename README.md => README.zh.md (95%) diff --git a/README.en.md b/README.en.md index 78d5903..856bfd6 100644 --- a/README.en.md +++ b/README.en.md @@ -12,7 +12,7 @@ Compass-CI monitors git repos of a large amount of open source software. Once Co **Login to the Test Environment** -The SSH is used to log in to the test environment for commissioning. +The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. **Test Result Analysis** @@ -46,7 +46,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate 1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. -3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. +3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. ## Contributing to Compass-CI @@ -66,4 +66,4 @@ Together, we can enhance the capabilities of - git bisect - Data analytics -- Data analytics result visualization \ No newline at end of file +- Data analytics result visualization diff --git a/README.md b/README.zh.md similarity index 95% rename from README.md rename to README.zh.md index b3ea5e4..5d02b49 100644 --- a/README.md +++ b/README.zh.md @@ -15,7 +15,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **调测环境登录** -使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… **测试结果分析** @@ -50,7 +50,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， 1. 向 compass-ci(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… -3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… +3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… ## Contributing to Compass-CI diff --git "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" index d70136f..dac74b6 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" @@ -57,7 +57,6 @@ ```shell hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - 2020-11-26 14:44:25 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait submit borrow-1h.yaml, got job_id=z9.170593 query=>{"job_id":["z9.170593"]} connect to ws://172.168.131.2:11310/filter @@ -154,7 +153,6 @@ ```shell hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - 2020-11-26 14:49:13 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait submit borrow-1h.yaml, got job_id=z9.170594 query=>{"job_id":["z9.170594"]} connect to ws://172.168.131.2:11310/filter diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index 5514de0..b59431e 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -2,9 +2,9 @@ # 1. 前提条件： ## 请先学习: -* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 -* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 -* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 +* [配置个人邮箱](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/ap… +* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如… +* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… # 2. 操作方法 diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index b986626..5a02431 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -2,7 +2,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 -:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +:exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email - receive email from compass-ci(a)qq.com -- 2.23.0

1 0

[PATCH v4 compass-ci] 优化部分文档
by Li Ping 03 Dec '20

03 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.en.md | 6 +++--- README.md | 4 ++-- ...\263\350\257\267\346\265\213\350\257\225\346\234\272.md" | 2 -- ...\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 6 +++--- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 5 files changed, 9 insertions(+), 11 deletions(-) diff --git a/README.en.md b/README.en.md index 78d5903..856bfd6 100644 --- a/README.en.md +++ b/README.en.md @@ -12,7 +12,7 @@ Compass-CI monitors git repos of a large amount of open source software. Once Co **Login to the Test Environment** -The SSH is used to log in to the test environment for commissioning. +The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. **Test Result Analysis** @@ -46,7 +46,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate 1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. -3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. +3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. ## Contributing to Compass-CI @@ -66,4 +66,4 @@ Together, we can enhance the capabilities of - git bisect - Data analytics -- Data analytics result visualization \ No newline at end of file +- Data analytics result visualization diff --git a/README.md b/README.md index b3ea5e4..5d02b49 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **调测环境登录** -使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… **测试结果分析** @@ -50,7 +50,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， 1. 向 compass-ci(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… -3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… +3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… ## Contributing to Compass-CI diff --git "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" index d70136f..dac74b6 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" @@ -57,7 +57,6 @@ ```shell hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - 2020-11-26 14:44:25 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait submit borrow-1h.yaml, got job_id=z9.170593 query=>{"job_id":["z9.170593"]} connect to ws://172.168.131.2:11310/filter @@ -154,7 +153,6 @@ ```shell hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - 2020-11-26 14:49:13 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait submit borrow-1h.yaml, got job_id=z9.170594 query=>{"job_id":["z9.170594"]} connect to ws://172.168.131.2:11310/filter diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index 5514de0..b59431e 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -2,9 +2,9 @@ # 1. 前提条件： ## 请先学习: -* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 -* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 -* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 +* [配置个人邮箱](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/ap… +* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如… +* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… # 2. 操作方法 diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index b986626..5a02431 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -2,7 +2,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 -:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +:exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email - receive email from compass-ci(a)qq.com -- 2.23.0

2 2

[PATCH v4 lkp-tests] ispec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 88 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 +++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 596 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..ba732182 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..4b6bbacc --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..9476bf3c --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..e1f2bbf6 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH lkp-tests] jobs: rm my_email field and set "pub_key" field as level-1 parameter in borrow-*.yaml
by Zhang Yale 03 Dec '20

03 Dec '20

[Why] my_email field is seted in .config/compass-ci/default/account.yaml, and now the "pub_key" field is not placed under the "sshd" field in yaml. Use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-10d.yaml | 15 +++++++-------- jobs/borrow-1d.yaml | 16 +++++++--------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/jobs/borrow-10d.yaml b/jobs/borrow-10d.yaml index aeb55466..a2b50025 100644 --- a/jobs/borrow-10d.yaml +++ b/jobs/borrow-10d.yaml @@ -1,14 +1,13 @@ suite: borrow testcase: borrow +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: # sleep at the bottom sleep: 10d diff --git a/jobs/borrow-1d.yaml b/jobs/borrow-1d.yaml index 72949982..7767f2d7 100644 --- a/jobs/borrow-1d.yaml +++ b/jobs/borrow-1d.yaml @@ -3,15 +3,13 @@ testcase: borrow timeout: 1d +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: - # sleep at the bottom sleep: 1d -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail_job_result.rb: fix NoMethodError
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] /home/lwt/compass-ci/lib/mail_job_result.rb:45:in `query_job': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 29a9c1a..50e2f53 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -33,7 +33,7 @@ class MailJobResult def set_submitter_info job = query_job - exit unless job['email'] + exit unless job && job['email'] @submitter_email = job['email'] @result_root = job['result_root'] @@ -42,6 +42,11 @@ class MailJobResult def query_job es = ESQuery.new query_result = es.multi_field_query({ 'id' => @job_id }) + if query_result['hits']['hits'].empty? + warn "Non-existent job: #{@job_id}" + return nil + end + query_result['hits']['hits'][0]['_source'] end end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 03 Dec '20

03 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..fe9545f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci] doc: change file name nr_threads->nr_threads.md
by Xu Xijian 03 Dec '20

03 Dec '20

Specify it as a markdown file. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/{nr_threads => nr_threads.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/job/{nr_threads => nr_threads.md} (100%) diff --git a/doc/job/nr_threads b/doc/job/nr_threads.md similarity index 100% rename from doc/job/nr_threads rename to doc/job/nr_threads.md -- 2.23.0

1 0

[PATCH v2 compass-ci] container: change vmlinuz path
by Xu Xijian 03 Dec '20

03 Dec '20

Since vmlinuz path changed from $rootfs_dir/vmlinuz to $rootfs_dir/boot/vmlinuz, here keep consistent. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/lib/common.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/update-os-docker/lib/common.rb b/container/update-os-docker/lib/common.rb index 23c4fdc..5b1353d 100755 --- a/container/update-os-docker/lib/common.rb +++ b/container/update-os-docker/lib/common.rb @@ -27,7 +27,7 @@ end def check_argv(argv) usage(argv) rootfs_dir = Pathname.new(OS_PATH + argv[0]).realpath.to_s - raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/vmlinuz') + raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/boot/vmlinuz') return rootfs_dir end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: set ENV DOCKER_REGISTRY_HOST default value
by Liu Yinsi 03 Dec '20

03 Dec '20

$DOCKER_REGISTRY_HOST default value should keep consistent with sparrow/3-code/dev-env, use server ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2753b9d..74905f1 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -26,7 +26,9 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF -: ${DOCKER_REGISTRY_HOST:="127.0.0.1"} +server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + +: ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} cat > /etc/docker/daemon.json <<EOF { -- 2.23.0

1 0

[PATCH v3 compass-ci] answerback-email: add execute tool limit
by Luan Shengde 03 Dec '20

03 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when execute answerback-email, the account info will be stored to the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 31 +++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..53c321d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,32 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'z9' + + message = 'please run the tool on z9 server' + puts message + + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +263,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 03 Dec '20

03 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..3827ea8 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = '' @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,25 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: { + match_all: {} + } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 91 +++++++++++++------ 1 file changed, 65 insertions(+), 26 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (56%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 56% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..22da3c5 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,22 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务 + 可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前sleep, + 并免密登录进去，手动输入命令或脚本进行调试，以host-info.yaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +24,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,31 +32,42 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： - 效果如下： +### 在测试任务运行失败时进入测试机调测任务 + 可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, + 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： -```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 - query=>{"job_id":["crystal.146528"]} - connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" - {"job_id": "crystal.146528", "job_state": "boot"} - {"job_id": "crystal.146528", "job_state": "download"} - "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} - {"job_state":"running","job_id":"crystal.146528"} - {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} +```yaml + suite: iperf + category: benchmark - root@vm-2p8g ~# + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +## 2.2 如何免密登录执行机？ +### 直接submit job.yaml或者submit加-m参数 + 可在收到的邮件里，按照邮件提示手动免密登录到执行机调测 + + 命令：submit iperf.yaml或者submit -m iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell @@ -84,3 +101,25 @@ root@vm-2p8g ~# ``` + +### submit job.yaml时加上-m -c + 该方法不用查看邮件，可自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` -- 2.23.0

2 3

[PATCH compass-ci] container: change vmlinuz path
by Xu Xijian 03 Dec '20

03 Dec '20

Since vmlinuz path changed from $rootfs_dir/vmlinuz to rootfs_dir/boot/vmlinuz, here keep consistent. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/lib/common.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/update-os-docker/lib/common.rb b/container/update-os-docker/lib/common.rb index 23c4fdc..5b1353d 100755 --- a/container/update-os-docker/lib/common.rb +++ b/container/update-os-docker/lib/common.rb @@ -27,7 +27,7 @@ end def check_argv(argv) usage(argv) rootfs_dir = Pathname.new(OS_PATH + argv[0]).realpath.to_s - raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/vmlinuz') + raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/boot/vmlinuz') return rootfs_dir end -- 2.23.0

2 2

[PATCH v2 compass-ci] answerback-email: add execute tool limit
by Luan Shengde 03 Dec '20

03 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when call es_client to store the account data, the data will be stored in the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 31 +++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..4c34e8f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,32 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'crystal.ci' + + message = 'please run the tool on z9 server' + + puts message + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +263,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..014600a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,29 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each { |tbox_agg| testboxes << tbox_agg['key'] } + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… [now values] - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/lib/job.cr | 8 ++++---- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2a811cb..a4345e0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -413,7 +413,7 @@ class Job end private def get_initrds - temp_initrds = [] of String + temp_initrds = [] of String if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) @@ -437,13 +437,13 @@ class Job end private def set_initrds_uri - uris = "" + initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - uris += "initrd #{initrd}\n" + initrds_uri_values << JSON::Any.new("initrd #{initrd}") end - self["initrds_uri"] = uris + @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) end private def set_user_lkp_src diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..3b68b59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -134,7 +134,7 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += job.initrds_uri + response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" response += job.kernel_uri response += job.kernel_params response += "\nboot\n" -- 2.23.0

1 0

[PATCH v3 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… [now values] - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/lib/job.cr | 6 ++++-- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2a811cb..ae3ba6e 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -413,7 +413,7 @@ class Job end private def get_initrds - temp_initrds = [] of String + temp_initrds = [] of String if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) @@ -438,12 +438,14 @@ class Job private def set_initrds_uri uris = "" + initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| uris += "initrd #{initrd}\n" + initrds_uri_values << JSON::Any.new("initrd #{initrd}") end - self["initrds_uri"] = uris + @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) end private def set_user_lkp_src diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..3b68b59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -134,7 +134,7 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += job.initrds_uri + response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" response += job.kernel_uri response += job.kernel_params response += "\nboot\n" -- 2.23.0

2 1

[PATCH v3 lkp-tests] spec/submit: add job yaml for changing multi-args
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..851f0473 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|whetstone-double + - 3|shell1 unixbench: -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: optimize regular matching in format_string function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index da8d8f3..03bf0b8 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -50,10 +50,8 @@ class Job private def format_string(original_str) temp = [] of String - original_str.each_char do |char| - if "#{char}" =~ /-|\.|\/|\w/ - temp << "#{char}".gsub("\/", "-") - end + original_str.gsub('/', '-').each_char do |char| + temp << "#{char}" if "#{char}" =~ /\w|-|\./ end return temp.join() -- 2.23.0

1 0

[PATCH compass-ci] sched: improve format string function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index da8d8f3..03bf0b8 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -50,10 +50,8 @@ class Job private def format_string(original_str) temp = [] of String - original_str.each_char do |char| - if "#{char}" =~ /-|\.|\/|\w/ - temp << "#{char}".gsub("\/", "-") - end + original_str.gsub('/', '-').each_char do |char| + temp << "#{char}" if "#{char}" =~ /\w|-|\./ end return temp.join() -- 2.23.0

2 4

[PATCH compass-ci] How to add a machine to compass-ci
by Zhang Dewan 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- ...00\350\210\254\350\277\207\347\250\213.md" | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 "doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" diff --git "a/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" new file mode 100644 index 0000000..3fa402b --- /dev/null +++ "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" @@ -0,0 +1,49 @@ + +这个过程可以使用“注册--发现”的模型来理解。 + +## 一台机器成为测试机: +- 需要完成注册 +- 需要能够执行调度器（scheduler）给它的任务 + +## 注册 +### 1. 注册到哪里 +测试机需要注册到z9上面的redis-db. + +x.x.x.x:xxxx> `hgetall "sched/mac2host"` # 登录redis, 这个命令可以查看已经注册的测试机信息 + +### 2. 怎么注册 +使用scheduler服务接口注册，模板： +`curl -X PUT "http://$server_ip:$port/set_host_mac?hostname=$hostname&mac=$mac` + +可以看到，这个调用需要两个参数：mac和hostname + +#### 2.1 接口参数说明 +##### 2.1.1 mac +当前我们的机器都是局域网内的资源，所以没有比使用mac注册更直接、方便的了。 + +###### 2.1.1.1 如何获取mac地址，这里用TaiShan服务器举例： +- 登录bmc，打开信息--系统信息--网络--网卡--第一张网卡--port1，即可查看。 +- 登录bios，进入Advanced--Network Device List，即可查看。 +- 使用一台和机器的bmc同网段的机器，执行以下命令即可查看： +`ipmitool -I lanplus -H $cur_ip -U $iBMC_user -P $iBMC_passwd raw 0x30 0x90 0x01 0x03 0x0${nic_idx} 0x00|awk '{print $5,$6,$7,$8,$9,$10}' |sed 's/\s/\-/g'` +- 给服务器安装os，登录上去，执行命令，即可查看： +`ip link | grep "^2:" -A 1| grep link |awk '{print $2}` + +##### 2.1.2 hostname +如同每一张网卡的mac是规则且唯一的，hostname作为资源的标识也是如此。 + +###### 2.1.2.1 如何定义hostname，这里还是以Taishan服务器为例： +` taishan200-2280-${cpuNum}s${coreNum}p-${memSize:-256}g--${aN}` +> taishan200-2280 服务器型号 +> cpuNum cpu个数 +> coreNum cpu核数 +> memSize 内存大小 +> aN 该类型机器编号 + +## 发现 +### 1. 怎么发现测试机并给它分配任务 +测试机器需要将启动方式设置pxe启动，然后在启动环节向scheduler发起请求，获取启动参数和任务信息。 + +### 2. 设置PXE启动 +- 华为服务器默认pxe启动，无需配置。 +- 其他设备，根据产品类型，按文档操作即可。 -- 2.23.0

2 2

[PATCH lab-z9] hosts: add host file of x86_64
by Zhang Yu 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/2288hv5-2s44p-384g | 4 ++++ hosts/2288hv5-2s44p-384g--b6 | 12 ++++++++++++ 2 files changed, 16 insertions(+) create mode 100644 hosts/2288hv5-2s44p-384g create mode 100644 hosts/2288hv5-2s44p-384g--b6 diff --git a/hosts/2288hv5-2s44p-384g b/hosts/2288hv5-2s44p-384g new file mode 100644 index 0000000..ffb5d45 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g @@ -0,0 +1,4 @@ +arch: x86_64 +nr_cpu: 88 +nr_node: 2 +memory: 384G diff --git a/hosts/2288hv5-2s44p-384g--b6 b/hosts/2288hv5-2s44p-384g--b6 new file mode 100644 index 0000000..2262524 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g--b6 @@ -0,0 +1,12 @@ +arch: x86_64 +nr_node: 2 +nr_cpu: 88 +memory: 384G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-MTFDDAK480TDC-1AT1ZABYY_20032653CE5E +mac_addr: + - 6c:eb:b6:50:c0:31 + - 6c:eb:b6:50:c0:32 +model_name: Intel(R) Xeon(R) Gold 6266C CPU @ 3.00GHz +ipmi_ip: 9.3.3.11 -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: update apply-account.md
by Luan Shengde 02 Dec '20

02 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/apply-account.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 362ead2..c1b5206 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -16,6 +16,7 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. +- the email name should be in english, as: "David Rientjes" in the template. - my oss commit: a git commit URL that has your contribution. ## step 2: receive an email @@ -36,3 +37,11 @@ Which contains account information for you: ## step 4: try it out Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + +## notice: + +In case that you received a failed email with message as below: + + The repo url for your commit is not in our upstream-repo list. + +Try to [register repo to upstream-repos](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: update apply-account.md
by Luan Shengde 02 Dec '20

02 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/apply-account.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 362ead2..4b18428 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -16,6 +16,7 @@ Email template: --- - please replace the 3 parts in {{ }} with your information. +- the email name should be in english, as: "David Rientjes" in the template. - my oss commit: a git commit URL that has your contribution. ## step 2: receive an email @@ -36,3 +37,11 @@ Which contains account information for you: ## step 4: try it out Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + +## notice: + +In case of you receive failed email with message: + + The repo url for your commit is not in our upstream-repo list. + +Try to [register repo to upstream-repos](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/m… -- 2.23.0

3 4

[PATCH compass-ci] lib/compare_data_format.rb: keep the size of x_params and dimension_values consistent
by Lu Kaiyi 02 Dec '20

02 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 484b186..1959045 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -49,12 +49,14 @@ class FormatEchartData def assign_echart_source(metric, value_type, dimensions) source = [] - source << @x_params + source << @x_params.clone dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| if @metrics_compare_results[x_param][metric] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + else + source[0].delete(x_param) end end source << dimension_values -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 86 ------------------- ...03\346\265\213\344\273\273\345\212\241.md" | 72 ++++++++++++++++ 2 files changed, 72 insertions(+), 86 deletions(-) delete mode 100644 doc/manual/how-to-log-in-the-machine.md create mode 100644 "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md deleted file mode 100644 index fefd718..0000000 --- a/doc/manual/how-to-log-in-the-machine.md +++ /dev/null @@ -1,86 +0,0 @@ -这篇文档将告诉你如何登陆测试环境 - -# 1. 前提条件 -请先学习: -* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 -* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 -* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 - -# 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： - -```shell - suite: host-info - category: functional - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: 1h - - host-info: -``` - -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： - 效果如下： - -```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 - query=>{"job_id":["crystal.146528"]} - connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" - {"job_id": "crystal.146528", "job_state": "boot"} - {"job_id": "crystal.146528", "job_state": "download"} - "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} - {"job_state":"running","job_id":"crystal.146528"} - {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} - - root@vm-2p8g ~# -``` - -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml - 任务执行完成后，系统发送邮件内容如下： - -```shell - Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use - - Dear $my_username: - Thanks for your participation in software ecosystem! - According to your application, vm-2p8g-294828 has been provisioned. - The datails are as follows: - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - Regards - Compass-Ci -``` - - 可通过ssh方式登陆到测试环境： - 命令： ssh root(a)api.compass-ci.openeuler.org -p $port - 效果如下： - -```shell - hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 - - root@vm-2p8g ~# -``` diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" new file mode 100644 index 0000000..1b962fe --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -0,0 +1,72 @@ +这篇文档将告诉你如何登陆测试环境去调测任务 + +# 1. 前提条件： +## 请先学习: +* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 +* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 +* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 + +# 2. 操作方法 + +## 2.1 定义免密登录的yaml + +### 方法一：在测试机运行脚本之前进入测试机调测任务 + +在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前sleep, 并免密登录进去，手动输入命令或脚本进行调试，以host-info.yaml为例： + +```yaml + suite: host-info + category: functional + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + sshd: + # sleep at the bottom + sleep: 1h + + host-info: +``` + +### 方法二：在测试任务运行失败时进入测试机调测任务 + +在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段，测试任务失败后，免密登录进去手动调试，以host-info.yaml为例： + +```yaml + suite: host-info + category: functional + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + host-info: + on_fail: + sshd: + sleep: 1000s +``` + +## 2.2 提交job.yaml + +### 方法一：直接提交 + + submit host-info.yaml + +在收到邮件后，按照邮件提示手动免密登录到执行机调测 + + ssh root(a)api.compass-ci.openeuler.org -p $port + +### 方法二：带参数提交 + + submit -m -c host-info.yaml + +该方法不用查看邮件，可自动免密登录到测试机 -- 2.23.0

1 0

[PATCH v2 compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..57cd5c1 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,15 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + return tbox_group.gsub(/(--|\.).*$/, '') +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

2 3

[PATCH compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..a897b57 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 02 Dec '20

02 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..c788607 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic configurations, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: memory size of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..4c99023 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = "" @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,23 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: {match_all: {} } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 02 Dec '20

02 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..cde2f0ed 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

2 2

[PATCH compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..6d5c47b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,17 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + tbox_group = tbox_group.split('--')[0] if tbox_group.include?('--') + tbox_group = tbox_group.split('.')[0] if tbox_group.include?('.') + return tbox_group +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

3 4

[PATCH compass-ci] sched: no need to check kernel config in initialized? function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7758e78..2a811cb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,8 +146,8 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel() set_depends_initrd() + set_kernel() set_initrds_uri() set_lkp_server() set_sshr_info() @@ -345,7 +345,11 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid"] + initialized_keys -= ["my_uuid", + "kernel_version", + "kernel_uri", + "kernel_params", + "linux_vmlinuz_path"] initialized_keys.each do |key| return false unless @hash.has_key?(key) -- 2.23.0

1 0

[PATCH compass-ci] auto_submit.yaml: change git_repo
by Li Yuanchao 02 Dec '20

02 Dec '20

To keep style the same Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index f741b79..e0f52ba 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,6 +1,6 @@ -AvxToNeon/AvxToNeon: +a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml -compass-ci/compass-ci: +c/compass-ci/compass-ci: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/start: add /etc/localtime
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index 01141c3..d7b1097 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -13,6 +13,7 @@ cmd=( --restart=always -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ result-webdav -- 2.23.0

1 0

[PATCH compass-ci] answerback-email: add execute tool limit
by Luan Shengde 02 Dec '20

02 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when call es_client to store the account data, the data will be stored in the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 33 +++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..77a2c1f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,34 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'crystal.ci' + + message = 'please run the tool on z9 server' + + puts message + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + puts 'check_erver' + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + puts 'check_my_name' + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +265,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 1

[PATCH v4 compass-ci] container/result-webdav: adjust the value of second_dir
by Cui Yili 02 Dec '20

02 Dec '20

[why] 1. when the uri starts with "//", etc., the previous regular expression did not get the second level directory correctly. 2. when the uploaded data package does not meet the format of "xxxx_yyyymmdd.cgz", uploading is also allowed, but no symbolic link. [how] 1. modify the variable second_dir regular expression. 2. make conditional judgments on link_name. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index c880522..98382bd 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,17 +54,21 @@ http { path = string.match(uri,"%g*/") upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/.+/(.*)/.+") + second_dir = string.match(uri, "/*%w+/(.-)/.+") - if (second_dir == "pkg") + if (second_dir == "pkg" or second_dir == "build-pkg") then - link_name = "latest.cgz" + link_name = "latest" else - link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + link_name = string.match(upload_file, "(.*)_%d+%.cgz") + end + + if (link_name ~= '') + then + link_name = link_name..".cgz" + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) end - - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) } } -- 2.23.0

1 0

[PATCH v4 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 02 Dec '20

02 Dec '20

Find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 418e7ff..b2ea831 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -21,6 +21,9 @@ INITRD_HTTP_PORT = (ENV.has_key?("INITRD_HTTP_PORT") ? ENV["INITRD_HTTP_PORT"] : OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1") OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 +CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") +LAB_REPO = "lab-z9" + SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 74ae4b2..a9eb3a7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -147,12 +147,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH lkp-tests] scheduler: fix: $server not $direct_server_ips
by Zhang Yu 02 Dec '20

02 Dec '20

[why] Now the server ip is defined by scheduler is $server not $direct_server_ips [how] Change the $direct_server_ips to $server Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 106ee0b9..1d424aaa 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -19,7 +19,7 @@ # db_driver : "${mysql_user:=root}" -: "${mysql_host:=$direct_server_ips}" +: "${mysql_host:=$server}" : "${mysql_port:=$mysql_port}" : "${mysql_db:=sysbench_1}" : "${mysql_password:=$mysql_password}" -- 2.23.0

1 0

[PATCH lab-z9] hosts: add taishan200-2280-2s48p-256g--a12 host-info
by Xiao Shenwei 02 Dec '20

02 Dec '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a12 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a12 diff --git a/hosts/taishan200-2280-2s48p-256g--a12 b/hosts/taishan200-2280-2s48p-256g--a12 new file mode 100644 index 0000000..ce014c1 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a12 @@ -0,0 +1,22 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000c500c276c347 + - /dev/disk/by-id/scsi-35000c500c276eb87 + - /dev/disk/by-id/scsi-35000c500c276b6ff +rootfs_partition: + - /dev/disk/by-id/scsi-35000c500c276f47b +mac_addr: + - 44:67:47:97:22:37 + - 44:67:47:97:22:38 + - 44:67:47:97:22:39 + - 44:67:47:97:22:3a + - 44:67:47:97:22:3b + - 44:67:47:97:22:3c + - 44:67:47:97:22:3d + - 44:67:47:97:22:3e +model_name: Kunpeng-920 +ipmi_ip: 9.3.5.2 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 02 Dec '20

02 Dec '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit: add job yaml for changing multi-args style
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 ++--- .../unixbench-spec-1-300-pipe.yaml | 25 +++++++++++++++++++ ...unixbench-spec-1-300-whetstone-double.yaml | 8 +++--- .../unixbench-spec-3-300-shell1.yaml | 8 +++--- .../unixbench-spec-3-300-shell8.yaml | 25 +++++++++++++++++++ 5 files changed, 61 insertions(+), 11 deletions(-) create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml create mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..34fb4996 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|pipe + - 3|shell8 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml new file mode 100644 index 00000000..44be6b10 --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-pipe.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: pipe +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: pipe + nr_task: '1' + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml index 9896bcef..78f79fa7 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '1' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: whetstone-double nr_task: '1' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml index 912d10ee..afe5a953 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '3' @@ -13,7 +13,7 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +21,5 @@ pp: runtime: 300 test: shell1 nr_task: '3' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml new file mode 100644 index 00000000..fa58377c --- /dev/null +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell8.yaml @@ -0,0 +1,25 @@ +--- +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell8 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell8 + nr_task: '3' + cpufreq_governor: +monitors: -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 69 ++++++++++++++----- 1 file changed, 53 insertions(+), 16 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (55%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 55% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..f4f33a3 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,20 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务，可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前就sleep, 并免密登录进去，可以手动输入命令或脚本进行调试，以host-inoyaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +22,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,19 +30,51 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： +### 在测试任务运行失败时进入测试机调测任务，可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： + +```yaml + suite: iperf + testcase: iperf + category: benchmark + + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp + + on_fail: + sshd: + sleep: 1000s +``` +## 如何免密登录测试机? +### submit时加上-m -c，该方法不用查看邮件登录，自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + 效果如下： ```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 query=>{"job_id":["crystal.146528"]} connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" {"job_id": "crystal.146528", "job_state": "boot"} {"job_id": "crystal.146528", "job_state": "download"} "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} @@ -48,9 +84,10 @@ root@vm-2p8g ~# ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +### 直接submit或者submit只加-m参数，可在收到的邮件里，按照邮件提示手动免密登录到测试机调测： + + 命令：submit iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell -- 2.23.0

1 0