- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/job.rb: delete matrix args
by Wei Jihui 04 Dec '20

04 Dec '20

if user add matrix args by overrides args, like: submit xxx.yaml 'matrix_key=matrix_value' ... it should delete matrix args from overrides, or it will keep matrix args in job.yaml all the time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2869dcb3..32329e10 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -302,6 +302,7 @@ class Job hash_job[key_array[i]] = v_array[i] end … [View More]

2 1

[PATCH lkp-tests] lib: delete the space on the left and right sides of the vertical bar
by Bai Jing 04 Dec '20

04 Dec '20

[why] The multi-args is like this "test|nr_threads", if user add it like this "test |nr_threads" or the value like "- shell1 | 1 ". So we need to delete the space Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2869dcb3..47fd56d3 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -290,11 +290,11 @@ class Job hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ - … [View More]

2 1

[PATCH compass-ci 3/3] scheduler/Sched: use JSONLogger in Sched class
by Wu Zhende 04 Dec '20

04 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 +-- src/scheduler/find_job_boot.cr | 6 +++--- src/scheduler/submit_job.cr | 2 +- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 22e0e4e..abb8a33 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -34,8 +34,7 @@ class Sched @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env - @log = JSONLogger.new -… [View More]

1 0

[PATCH compass-ci 1/3] lib/json_logger.cr: support initialize env
by Wu Zhende 04 Dec '20

04 Dec '20

Can use JSONLogger.new(env: env) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 2b5ef6a..d136bdd 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -6,8 +6,7 @@ require "json" require "any_merge" class JSONLogger < Logger - def initialize(logdev = STDOUT, formatter = my_formatter) - @env = nil + def … [View More]

1 0

[PATCH compass-ci] answerback-email: add check account exists
by Luan Shengde 04 Dec '20

04 Dec '20

add check account exists for assigning/updating account avoid re-assign accounts for single email address [why] the tool may assign a new account for the email even if the email already has one. It will cause a big waste of account resource for wrong operation or test. [how] add check account exists for the email before assigning a new account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 38 ++++++++++++++------ 1 file changed, 27 … [View More]insertions(+), 11 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 53c321d..836ad86 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -61,21 +61,19 @@ conf_info = { 'is_update_account' => false } -def init_info(mail_content, email_info, my_info) +def init_info(mail_content, email_info, my_info, my_info_es) my_info['my_email'] = mail_content.from[0] + + read_my_login_name(my_info['my_email'], my_info_es) email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - return if mail_content.attachments.empty? - email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip || nil end def read_my_login_name(my_email, my_info_es) - my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) - my_account_info = YAML.safe_load my_account_info_str - message = "No such email found from the ES: #{my_email}" - raise message unless my_account_info['found'] + my_account_info = ESQuery.new(index: 'accounts').query_by_id(my_email) - my_info_es.update my_account_info['_source'] + my_info_es.update my_account_info || {} end options = OptionParser.new do |opts| @@ -102,6 +100,7 @@ options = OptionParser.new do |opts| end my_info['my_email'] = email_address + read_my_login_name(email_address, my_info_es) end opts.on('-n name', '--name name', 'appoint name') do |name| @@ -134,7 +133,7 @@ options = OptionParser.new do |opts| return false end mail_content = Mail.read(email_file) - init_info(mail_content, email_info, my_info) + init_info(mail_content, email_info, my_info, my_info_es) end opts.on('-g', '--gen-sshkey', 'generate jumper rsa public/private key and return pubkey') do @@ -142,7 +141,6 @@ options = OptionParser.new do |opts| end opts.on('-u', '--update', 'updata configurations') do - read_my_login_name(my_info['my_email'], my_info_es) conf_info['is_update_account'] = true end @@ -223,7 +221,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == 'crystal.ci' message = 'please run the tool on z9 server' puts message @@ -240,9 +238,27 @@ def check_my_name_exist(my_info) return false end +def check_email_assigned_account(conf_info, my_info_es) + if conf_info['is_update_account'] + return true unless my_info_es.empty? + + message = "The email has not assigned an account yet.\n" + message += "Delete the '-u' option to assign a new account." + else + return true if my_info_es.empty? + + message = "The email has already assigned an account.\n" + message += "You can use option '-u' to update the account." + end + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_server return unless check_my_email(my_info) + return unless check_email_assigned_account(conf_info, my_info_es) my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0 [View Less]

1 0

[PATCH v4 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assign accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - … [View More]my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 34 +++++++++++++++++------ 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..99da510 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,30 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) end def store_account_info -- 2.23.0 [View Less]

1 0

[PATCH v3 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assigned accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - … [View More]my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..0b94ca5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,32 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) + # my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + # YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0 [View Less]

1 1

[PATCH v2 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid multi accounts assignment for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when get the request for applying account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name -… [View More] my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..d499840 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,6 +12,7 @@ require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +ES_PORT = ENV['ES_PORT'] || 9200 # assign uuid/account for user # when mail-robot listened new email, and the email's subject @@ -51,6 +52,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +64,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +84,31 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es['found'] + apply_info.update my_account_es['_source'] + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es['found'] + end + + def check_account_es + my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0 [View Less]

2 2

[PATCH v3 lkp-tests] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

[why] original iozone tool can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra whitespace after 2892445): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/… [View More]

1 0

[PATCH lkp-tests] Automatic compilation and configure the variables
by Hu Xuejiao 04 Dec '20

04 Dec '20

[why] Source shell_file based on different shells. [how] First, make and make install is written to the make-env. Second, invoking shell_profile(env.sh) that will distinguish different shell_files. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- make-env | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 make-env diff --git a/make-env b/make-env new file mode 100644 index 00000000..cab39288 --- /dev/null +++ b/make-env @@ -0,0 +1,6 @@ +#!/bin/bash +. lib/env.sh +make +… [View More]

3 2

[PATCH compass-ci] git_mirror.rb: disable some unnecessary messages
by Li Yuanchao 04 Dec '20

04 Dec '20

Disable some unnecessary messages, so important logs will not be covered. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index edee331..9f254f5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -235,7 +235,7 @@ class MirrorMain return if @git_info[git_repo]['is_submodule'] mirror_dir = "/srv/git/#{git_repo}.git" - show_ref_out = %x(git … [View More]

1 0

[PATCH compass-ci] auto_submit.yaml: change testbox of deploy-cci
by Li Yuanchao 04 Dec '20

04 Dec '20

'vm-2p8g' is not enough for self test. It's suggested physical machine is better. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index e0f52ba..b53b647 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,8 +1,8 @@ a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=… [View More]

1 0

[PATCH lkp-tests] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.… [View More]

2 1

[PATCH compass-ci] sparrow: add mail-robot work dir, ENV, permission
by Liu Yinsi 04 Dec '20

04 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 3 +++ sparrow/3-code/dev-env | 1 + 4 files changed, 7 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 74905f1..b381d73 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -5,6 +5,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch … [View More]

1 0

[PATCH v5 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 04 Dec '20

04 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- .../job_on_fail-3000.yaml => job-3000.yaml} | 0 .../{job_on_fail.yaml => job.yaml} | 1 + ...68000000-1-65535-set_get-never-never.yaml} | 0 .../redis.yaml => link_jobs1/job.yaml} | 0 .../job-1000-2-100000ss-add.yaml} | 0 .../job-10000-2-100000ss-add.yaml} | 0 .../job-20-2-6000ss-cache.yaml} … [View More] | 0 .../ftq.yaml => link_jobs2/job.yaml} | 0 ...e.yaml => job-1-300-whetstone-double.yaml} | 0 ...-300-shell1.yaml => job-3-300-shell1.yaml} | 0 .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ..._files-defaults.yaml => job-defaults.yaml} | 0 spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - ...-add.yaml => job-1000-2-100000ss-add.yaml} | 0 ...add.yaml => job-10000-2-100000ss-add.yaml} | 0 ...-cache.yaml => job-20-2-6000ss-cache.yaml} | 0 .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 30 +++++-------------- 19 files changed, 9 insertions(+), 24 deletions(-) rename spec/submit/job_on_fail/{job_on_fail/job_on_fail-3000.yaml => job-3000.yaml} (100%) rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (100%) rename spec/submit/{link_jobs/redis.yaml => link_jobs1/job.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-1000-2-100000ss-add.yaml => link_jobs2/job-1000-2-100000ss-add.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-10000-2-100000ss-add.yaml => link_jobs2/job-10000-2-100000ss-add.yaml} (100%) rename spec/submit/{link_jobs/ftq/ftq-20-2-6000ss-cache.yaml => link_jobs2/job-20-2-6000ss-cache.yaml} (100%) rename spec/submit/{link_jobs/ftq.yaml => link_jobs2/job.yaml} (100%) rename spec/submit/matrix/{unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml => job-1-300-whetstone-double.yaml} (100%) rename spec/submit/matrix/{unixbench-spec/unixbench-spec-3-300-shell1.yaml => job-3-300-shell1.yaml} (100%) rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) rename spec/submit/merge_yaml/{merge_included_files/merge_included_files-defaults.yaml => job-defaults.yaml} (100%) create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml rename spec/submit/separate_yaml/{ftq/ftq-1000-2-100000ss-add.yaml => job-1000-2-100000ss-add.yaml} (100%) rename spec/submit/separate_yaml/{ftq/ftq-10000-2-100000ss-add.yaml => job-10000-2-100000ss-add.yaml} (100%) rename spec/submit/separate_yaml/{ftq/ftq-20-2-6000ss-cache.yaml => job-20-2-6000ss-cache.yaml} (100%) rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml similarity index 100% rename from spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml rename to spec/submit/job_on_fail/job-3000.yaml diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs1/job.yaml similarity index 100% rename from spec/submit/link_jobs/redis.yaml rename to spec/submit/link_jobs1/job.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs2/job.yaml similarity index 100% rename from spec/submit/link_jobs/ftq.yaml rename to spec/submit/link_jobs2/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml rename to spec/submit/matrix/job-1-300-whetstone-double.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml rename to spec/submit/matrix/job-3-300-shell1.yaml diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml similarity index 100% rename from spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml rename to spec/submit/merge_yaml/job-defaults.yaml diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..c8cdcc15 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -16,20 +16,18 @@ end def traverse_file(output_dir) Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |output_dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", '-o', output_dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{output_dir}/job.yaml" ] system(*submit_cmd) traverse_file(output_dir) @@ -38,22 +36,8 @@ end describe 'submit job spec' do it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') + submit_job() end end + +submit_job -- 2.23.0 [View Less]

1 0

[PATCH compass-ci 2/2] lib/mail_bisect_result.rb: parse and send error_messages
by Cao Xueliang 04 Dec '20

04 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 2799ce2..5a208c4 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -7,7 +7,7 @@ require 'json' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @error_id = bisect_info['error_id'] + @… [View More]

1 0

[PATCH compass-ci 1/2] lib/git_bisect.rb: replace error_id with error_messages
by Cao Xueliang 04 Dec '20

04 Dec '20

The error_messages will display the original error messages. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 9033db2..e2c5636 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -4,6 +4,7 @@ require 'json' require_relative 'es_query' +require_relative 'error_messages' require_relative 'sched_client' require_relative '../src/… [View More]

1 0

[PATCH v2 lkp-tests 2/2] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.… [View More]

1 0

[PATCH v2 lkp-tests 1/2] stats/crystal: sysbench-threads-git to sysbench-threads-git.cr
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/sysbench-threads-git.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 stats/crystal/sysbench-threads-git.cr diff --git a/stats/crystal/sysbench-threads-git.cr b/stats/crystal/sysbench-threads-git.cr new file mode 100644 index 00000000..91707e87 --- /dev/null +++ b/stats/crystal/sysbench-threads-git.cr @@ -0,0 +1,14 @@ +#!/usr/bin/env crystal + +# General statistics: +# total time: … [View More]

1 0

[PATCH v2 compass-ci] How to add a machine to compass-ci
by Zhang Dewan 04 Dec '20

04 Dec '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> How a new machine add to compass-ci Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- ...00\350\210\254\350\277\207\347\250\213.md" | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 "doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" diff --git "a/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\… [View More]345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" new file mode 100644 index 0000000..340ffea --- /dev/null +++ "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" @@ -0,0 +1,54 @@ + +这个过程可以使用“注册--发现”的模型来理解。 + +## 一台机器成为测试机: +- 需要注册到compass-ci +- 需要能够执行调度器（scheduler）给它的任务 + +## 注册 +### 1. 注册信息保存在哪 +测试机需要注册到z9上面的redis-db. + +x.x.x.x:xxxx> `hgetall "sched/mac2host"` # 登录redis, 这个命令可以查看已经注册的测试机信息 + +### 2. 怎么注册 +使用scheduler服务接口注册，模板： +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}"` +`curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${api_queues}"` + +可以看到，这两个调用需要三个参数：mac、hostname、api_queues + +#### 2.1 接口参数说明 +##### 2.1.1 mac +当前我们的机器都是局域网内的资源，所以没有比使用mac注册更直接、方便的了。 + +###### 2.1.1.1 如何获取mac地址，这里用TaiShan服务器举例： +- 登录bmc，打开信息--系统信息--网络--网卡--第一张网卡--port1，即可查看。 +- 登录bios，进入Advanced--Network Device List，即可查看。 +- 使用一台和机器的bmc同网段的机器，执行以下命令即可查看： +`ipmitool -I lanplus -H $cur_ip -U $iBMC_user -P $iBMC_passwd raw 0x30 0x90 0x01 0x03 0x0${nic_idx} 0x00|awk '{print $5,$6,$7,$8,$9,$10}' |sed 's/\s/\-/g'` +- 给服务器安装os，登录上去，执行命令，即可查看： +`ip link | grep "^2:" -A 1| grep link |awk '{print $2}` + +##### 2.1.2 hostname +如同每一张网卡的mac是规则且唯一的，hostname作为资源的标识也是如此。 + +###### 2.1.2.1 如何定义hostname，这里还是以Taishan服务器为例： +` taishan200-2280-${cpuNum}s${coreNum}p-${memSize:-256}g--${aN}` +> taishan200-2280 服务器型号 +> cpuNum cpu个数 +> coreNum cpu核数 +> memSize 内存大小 +> aN 该类型机器编号 + +#### 2.1.3 api_queues +作为单台机器的组标识，一般取hostname的前缀。 +比如，taishan200-2280-2s64p-256g--a100, 它的api_queues为：taishan200-2280-2s64p-256g + +## 发现 +### 1. 怎么发现测试机并给它分配任务 +测试机器需要将启动方式设置pxe启动，然后在启动环节向scheduler发起请求，获取启动参数和任务信息。 + +### 2. 设置PXE启动 +- 华为服务器默认pxe启动，无需配置。 +- 其他设备，根据产品类型，按文档操作即可。 -- 2.23.0 [View Less]

2 1

[PATCH lkp-tests] stats/crystal: sysbench-threads-git to sysbench-threads-git.cr
by Lu Kaiyi 04 Dec '20

04 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/sysbench-threads-git.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 stats/crystal/sysbench-threads-git.cr diff --git a/stats/crystal/sysbench-threads-git.cr b/stats/crystal/sysbench-threads-git.cr new file mode 100644 index 00000000..5d9dcc10 --- /dev/null +++ b/stats/crystal/sysbench-threads-git.cr @@ -0,0 +1,14 @@ +#!/usr/bin/env crystal + +# General statistics: +# total time: … [View More]

2 1

[PATCH v2 compass-ci] user-client/maintain: add '|' for multi args
by Liu Yinsi 04 Dec '20

04 Dec '20

multi args changed from os os_arch os_version to os|os_arch|os_version, and if use os|os_arch|os_version, job yaml must as the first parameter. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ed97999..9f75516 100755 --- a/user-client/… [View More]

1 0

[PATCH compass-ci] change file mode to be the same with others
by Ren Wen 04 Dec '20

04 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/json_logger.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 src/lib/json_logger.cr diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] scheduler/jobfile_operate.cr: refactor create_job_cpio
by Wu Zhende 04 Dec '20

04 Dec '20

Separate "create_job_cpio" into a new file. Use class Sched, can use Sched's instance variable. Also can simplify jobfile_operate.cr. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/create_job_cpio.cr | 140 +++++++++++++++++++++++++++++++ src/scheduler/find_job_boot.cr | 2 +- src/scheduler/jobfile_operate.cr | 132 ----------------------------- 4 files changed, 142 insertions(+), 133 deletions(-) create mode 100644 src/… [View More]scheduler/create_job_cpio.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a9f893f..22e0e4e 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -20,6 +20,7 @@ require "../scheduler/find_next_job_boot" require "../scheduler/close_job" require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" +require "../scheduler/create_job_cpio" class Sched property es diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr new file mode 100644 index 0000000..5c1be7e --- /dev/null +++ b/src/scheduler/create_job_cpio.cr @@ -0,0 +1,140 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# shellwords require from '/c/lkp-tests/lib/' +require "shellwords" +require "file_utils" +require "json" +require "yaml" + +require "./jobfile_operate" + +class Sched + private def prepare_dir(file_path : String) + file_path_dir = File.dirname(file_path) + unless File.exists?(file_path_dir) + FileUtils.mkdir_p(file_path_dir) + end + end + + private def valid_shell_variable?(key) + key =~ /^[a-zA-Z_]+[a-zA-Z0-9_]*$/ + end + + private def create_job_sh(job_sh_content : Array(JSON::Any), path : String) + File.open(path, "w", File::Permissions.new(0o775)) do |file| + file.puts "#!/bin/sh\n\n" + + job_sh_content.each do |line| + if line.as_a? + line.as_a.each { |val| file.puts val } + else + file.puts line + end + end + + file.puts "\"$@\"" + end + end + + private def shell_escape(val) + val = val.join "\n" if val.is_a?(Array) + + if val.nil? || val.empty? + return nil + elsif val =~ /^[+-]?([0-9]*\.?[0-9]+|[0-9]+\.?[0-9]*)([eE][+-]?[0-9]+)?$/ + return val + elsif !val.includes?("'") && !val.includes?("$") + return "'#{val}'" + elsif !val.includes?('"') + return "\"#{val}\"" + else + return Shellwords.shellescape(val) + end + end + + private def parse_one(script_lines, key, val) + return false if val.as_h? || !valid_shell_variable?(key) + + value = shell_escape(val.as_a? || val.to_s) + script_lines << "\texport #{key}=" + value if value + end + + private def sh_export_top_env(job_content : Hash) + script_lines = ["export_top_env()", "{"] of String + + job_content.each { |key, val| parse_one(script_lines, key, val) } + + script_lines << "\n" + script_lines << "\t[ -n \"$LKP_SRC\" ] ||" + script_lines << "\texport LKP_SRC=/lkp/${user:-lkp}/src" + script_lines << "}\n\n" + + script_lines = script_lines.to_s + script_lines = JSON.parse(script_lines) + end + + def create_job_cpio(job_content : JSON::Any, base_dir : String) + job_content = job_content.as_h + + # put job2sh in an array + if job_content.has_key?("job2sh") + tmp_job_sh_content = job_content["job2sh"] + + job_sh_array = [] of JSON::Any + tmp_job_sh_content.as_h.each do |_key, val| + next if val == nil + job_sh_array += val.as_a + end + else + job_sh_array = [] of JSON::Any + end + + # generate job.yaml + temp_yaml = base_dir + "/#{job_content["id"]}/job.yaml" + prepare_dir(temp_yaml) + + # no change to <object> content { "#! jobs/pixz.yaml": null } + # - this will create a <'#! jobs/pixz.yaml':> in the yaml file + # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml + # tested : no effect to job.sh + File.open(temp_yaml, "w") do |file| + YAML.dump(job_content, file) + end + + # generate unbroken job shell content + sh_export_top_env = sh_export_top_env(job_content) + job_sh_content = sh_export_top_env.as_a + job_sh_array + + # generate job.sh + job_sh = base_dir + "/#{job_content["id"]}/job.sh" + create_job_sh(job_sh_content.to_a, job_sh) + + job_dir = base_dir + "/#{job_content["id"]}" + + if job_sh_array.empty? + lkp_src = Jobfile::Operate.prepare_lkp_tests(job_content["lkp_initrd_user"], + job_content["os_arch"]) + + cmd = "#{lkp_src}/sbin/create-job-cpio.sh #{temp_yaml}" + idd = `#{cmd}` + else + cmd = "./create-job-cpio.sh #{job_dir}" + idd = `#{cmd}` + end + + # if the create job cpio failed, what to do? + puts idd if idd.match(/ERROR/) + + # create result dir and copy job.sh, job.yaml and job.cgz to result dir + src_dir = File.dirname(temp_yaml) + dst_dir = File.join("/srv", job_content["result_root"].to_s) + FileUtils.mkdir_p(dst_dir) + + # the job.yaml is not final version + files = ["#{src_dir}/job.sh", + "#{src_dir}/job.yaml", + "#{src_dir}/job.cgz"] + FileUtils.cp(files, dst_dir) + end +end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ad48f27..6f51640 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -75,7 +75,7 @@ class Sched job = get_job_from_queues(queues, host) if job - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) end return boot_content(job, boot_type) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..df79d3b 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -5,139 +5,7 @@ require "file_utils" require "json" require "yaml" -# require from '/c/lkp-tests/lib/' -require "shellwords" - -LKP_PATH = "/c/lkp-tests" -if ENV["LKP_SRC"] != LKP_PATH - raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} #{LKP_PATH}" -end - module Jobfile::Operate - def self.prepare_dir(file_path : String) - file_path_dir = File.dirname(file_path) - unless File.exists?(file_path_dir) - FileUtils.mkdir_p(file_path_dir) - end - end - - def self.valid_shell_variable?(key) - key =~ /^[a-zA-Z_]+[a-zA-Z0-9_]*$/ - end - - def self.create_job_sh(job_sh_content : Array(JSON::Any), path : String) - File.open(path, "w", File::Permissions.new(0o775)) do |file| - file.puts "#!/bin/sh\n\n" - job_sh_content.each do |line| - if line.as_a? - line.as_a.each { |val| file.puts val } - else - file.puts line - end - end - file.puts "\"$@\"" - end - end - - def self.shell_escape(val) - val = val.join "\n" if val.is_a?(Array) - - if val.nil? || val.empty? - return nil - elsif val =~ /^[+-]?([0-9]*\.?[0-9]+|[0-9]+\.?[0-9]*)([eE][+-]?[0-9]+)?$/ - return val - elsif !val.includes?("'") && !val.includes?("$") - return "'#{val}'" - elsif !val.includes?('"') - return "\"#{val}\"" - else - return Shellwords.shellescape(val) - end - end - - def self.parse_one(script_lines, key, val) - return false if val.as_h? || !valid_shell_variable?(key) - - value = shell_escape(val.as_a? || val.to_s) - script_lines << "\texport #{key}=" + value if value - end - - def self.sh_export_top_env(job_content : Hash) - script_lines = ["export_top_env()", "{"] of String - - job_content.each { |key, val| parse_one(script_lines, key, val) } - - script_lines << "\n" - script_lines << "\t[ -n \"$LKP_SRC\" ] ||" - script_lines << "\texport LKP_SRC=/lkp/${user:-lkp}/src" - script_lines << "}\n\n" - - script_lines = script_lines.to_s - script_lines = JSON.parse(script_lines) - end - - def self.create_job_cpio(job_content : JSON::Any, base_dir : String) - job_content = job_content.as_h - - # put job2sh in an array - if job_content.has_key?("job2sh") - tmp_job_sh_content = job_content["job2sh"] - - job_sh_array = [] of JSON::Any - tmp_job_sh_content.as_h.each do |_key, val| - next if val == nil - job_sh_array += val.as_a - end - else - job_sh_array = [] of JSON::Any - end - - # generate job.yaml - temp_yaml = base_dir + "/#{job_content["id"]}/job.yaml" - prepare_dir(temp_yaml) - - # no change to <object> content { "#! jobs/pixz.yaml": null } - # - this will create a <'#! jobs/pixz.yaml':> in the yaml file - # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml - # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) - end - - # generate unbroken job shell content - sh_export_top_env = sh_export_top_env(job_content) - job_sh_content = sh_export_top_env.as_a + job_sh_array - - # generate job.sh - job_sh = base_dir + "/#{job_content["id"]}/job.sh" - create_job_sh(job_sh_content.to_a, job_sh) - - job_dir = base_dir + "/#{job_content["id"]}" - - if job_sh_array.empty? - lkp_src = prepare_lkp_tests(job_content["lkp_initrd_user"], - job_content["os_arch"]) - - cmd = "#{lkp_src}/sbin/create-job-cpio.sh #{temp_yaml}" - idd = `#{cmd}` - else - cmd = "./create-job-cpio.sh #{job_dir}" - idd = `#{cmd}` - end - - # if the create job cpio failed, what to do? - puts idd if idd.match(/ERROR/) - # create result dir and copy job.sh, job.yaml and job.cgz to result dir - src_dir = File.dirname(temp_yaml) - dst_dir = File.join("/srv", job_content["result_root"].to_s) - FileUtils.mkdir_p(dst_dir) - # the job.yaml is not final version - files = ["#{src_dir}/job.sh", - "#{src_dir}/job.yaml", - "#{src_dir}/job.cgz"] - FileUtils.cp(files, dst_dir) - end - def self.unzip_cgz(source_path : String, target_path : String) FileUtils.mkdir_p(target_path) cmd = "cd #{target_path};gzip -dc #{source_path}|cpio -id" -- 2.23.0 [View Less]

1 0

[PATCH lkp-tests 2/2] lib/common.rb: add build_pkg_error_id function
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/common.rb | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/lib/common.rb b/lib/common.rb index 91997baa..cc64eb17 100755 --- a/lib/common.rb +++ b/lib/common.rb @@ -319,3 +319,25 @@ end def delete_file_if_exist(file) File.delete(file) if File.exist?(file) end + +def build_pkg_error_id(line) + line = line.chomp + line = line.gsub(/\b[3-9]\.[0-9]+[-a-z0-9.]+/, "#") # linux version: 3.17.0-next-20141008-… [View More]

1 0

[PATCH lkp-tests 1/2] stats/build-pkg: rename common_error_id function and move it to ../lib/common
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 28 +++++----------------------- 1 file changed, 5 insertions(+), 23 deletions(-) diff --git a/stats/build-pkg b/stats/build-pkg index 9eef8c91..90f27656 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -1,27 +1,9 @@ #!/usr/bin/env ruby -require 'set' -def common_error_id(line) - line = line.chomp - line = line.gsub(/\b[3-9]\.[0-9]+[-a-z0-9.]+/, "#") # linux version: 3.17.0-next-20141008-g099669ed - line = … [View More]

1 0

[PATCH compass-ci] lib/error_messages.rb: add obtain_error_messages_by_errorid function
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 7bf3aa8..3a74d29 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -3,10 +3,13 @@ # frozen_string_literal: true require 'set' +require_relative "#{ENV['LKP_SRC']}/lib/common" # usage: # foo = ErrorMessages.new(build-pkg) # build-pkg is a file that includes errors generated by … [View More]

1 0

[PATCH compass-ci] user-client/maintain: add '|' for multi args
by Liu Yinsi 03 Dec '20

03 Dec '20

multi args changed from os os_arch os_version to os|os_arch|os_version, and if use os|os_arch|os_version, job yaml must as the first parameter. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ed97999..9f75516 100755 --- a/user-client/… [View More]

1 0

[PATCH lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 34 ++++++---------------------------- 1 file changed, 6 insertions(+), 28 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..75ad9798 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -16,44 +16,22 @@ end def traverse_file(output_dir) Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - … [View More]

1 0

[PATCH compass-ci] src/scheduler: delete job2sh field in job.yaml
by Zhang Yu 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/scheduler/jobfile_operate.cr | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..82ed885 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -80,16 +80,13 @@ module Jobfile::Operate job_content = job_content.as_h # put job2sh in an array - if job_content.has_key?("job2sh") - … [View More]

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix upload in container
by Wang Yong 03 Dec '20

03 Dec '20

docker can not use mount operation, so now use curl to upload build-pkg cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index cde2f0edd..38d8bace9 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -18,6 +18,8 @@ ## See: https://wiki.archlinux.org/index.php/Makepkg . $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/… [View More]

2 1

[PATCH v3 compass-ci] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..5dc68c8 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,28 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job … [View More]

1 0

[PATCH compass-ci] container/conserver: change executable script
by Cui Yili 03 Dec '20

03 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/build | 2 +- container/conserver/{conserver-head.cf => conserver.cf} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename container/conserver/{conserver-head.cf => conserver.cf} (100%) diff --git a/container/conserver/build b/container/conserver/build index 25c785c..0700c5e 100755 --- a/container/conserver/build +++ b/container/conserver/build @@ -2,6 +2,6 @@ # SPDX-License-… [View More]

1 0

[PATCH compass-ci] manual/如何申请测试机.md: change the borrow-1h.yaml to latest
by Zhang Yale 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...257\267\346\265\213\350\257\225\346\234\272.md" | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" index d70136f..85d385d 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\… [View More]

2 2

[PATCH compass-ci 2/2] mail-robot: update start
by Luan Shengde 03 Dec '20

03 Dec '20

add MAILDIR to default yaml file avoid change start file if there is a new maildir. add ES_PORT Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 1c55dd3..f8078b6 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -14,10 +14,11 @@ names = Set.new %w[ SEND_MAIL_PORT SCHED_HTTPS_HOST … [View More]

1 0

[PATCH compass-ci 1/2] mail-robot: add check assigned account for email
by Luan Shengde 03 Dec '20

03 Dec '20

avoid multi accounts assignment for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. this is a big waste of account resource [how] when get the request for applying account, check whether the email already has an account before applying account. apply a new one if there is none return the old one if there is already one for the email. and update my_info if the email has new: my_name my_ssh_pubkey Signed-off-… [View More]by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 37 +++++++++++++++++------ 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..24f41bc 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,6 +12,7 @@ require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +ES_PORT = ENV['ES_PORT'] || 9200 # assign uuid/account for user # when mail-robot listened new email, and the email's subject @@ -51,6 +52,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +64,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +84,32 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es['found'] + apply_info.update my_account_es['_source'] + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es['found'] + # return acct_info + end + + def check_account_es + my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0 [View Less]

1 0

[PATCH v2 compass-ci] doc/manual: update submit命令详解.md
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- ...bmit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 6 ++++++ 1 file changed, 6 insertions(+) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 0edb045..66d8cc4 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.… [View More]

1 0

[PATCH v2 lkp-tests] tests/build-pkg: unified indent
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 148 ++++++++++++++++++++++++------------------------ 1 file changed, 74 insertions(+), 74 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index cde2f0ed..b2679850 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -21,60 +21,60 @@ check_vars() { - [ -n "$os" ] || die "os is empty" - [ -n "$os_arch" ] || die "os_arch is empty" - [ -n "$os_version" ] || die "os_version is empty" - [ -n "$… [View More]pkgbuild_repo" ] || die "pkgbuild_repo is empty" - [ -n "$upstream_commit" ] || die "upstream_commit is empty" - [ -n "$os_mount" ] || die "os_mount is empty" + [ -n "$os" ] || die "os is empty" + [ -n "$os_arch" ] || die "os_arch is empty" + [ -n "$os_version" ] || die "os_version is empty" + [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" + [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { - # the same image is mounted to cifs and nfs, the generated cgz files - # are stored in the nfs directory. - [[ "$os_mount" = "cifs" ]] && os_mount="nfs" - pack_to=${os_mount}/${os}/${os_arch}/${os_version} + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} - PKG_MNT=/initrd/build-pkg - mkdir -p "$PKG_MNT" + PKG_MNT=/initrd/build-pkg + mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } } get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" -} + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" + } request_pkg() { - local pkgrepo="${pkgbuild_repo%%//*}" - local pkgpath="./" - [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" - - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') - - for pkgfile in ${filelist[*]} - do - get_pkgfile "${pkgrepo}" "$pkgfile" - done - - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" - } - - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" + local pkgrepo="${pkgbuild_repo%%//*}" + local pkgpath="./" + [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" + + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') + + for pkgfile in ${filelist[*]} + do + get_pkgfile "${pkgrepo}" "$pkgfile" + done + + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + + [ -f PKGBUILD ] || die "PKGBUILD not exist" + [ -s PKGBUILD ] || die "PKGBUILD is empty" } create_softlink() @@ -95,54 +95,54 @@ create_softlink() replace_source() { - sed -i "s|^source=[^)]*|$1|g" "$2" + sed -i "s|^source=[^)]*|$1|g" "$2" } # replace *.tar.* address from source field in PKGBUILD replace_tar_source() { - echo "source=(" > PKGBUILD.src - for url in ${source[@]} - do - echo "$url" | egrep '.+\.tar' && { - local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" - # check whether local server exists *.tar.* file - wget -q --spider "${local_url}" - [ "$?" == 0 ] || continue + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue - url="\"${local_url}\"" + url="\"${local_url}\"" } - echo "$url" >> PKGBUILD.src - done - echo ")" >> PKGBUILD.src + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src } build_source_pkg() { - local repo_dir="" - source PKGBUILD - [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { - - if [[ "$source" = *::* ]]; then - repo_dir="${source%%::*}::" - else - repo_dir="$(basename ${upstream_url})" - repo_dir="${repo_dir%.git*}::" - fi - - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" - replace_source "${upstream_source}" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" - } + local repo_dir="" + source PKGBUILD + [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { + + if [[ "$source" = *::* ]]; then + repo_dir="${source%%::*}::" + else + repo_dir="$(basename ${upstream_url})" + repo_dir="${repo_dir%.git*}::" + fi + + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" + } - replace_tar_source + replace_tar_source - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] && create_softlink } check_vars -- 2.23.0 [View Less]

1 0

[PATCH v5 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 49 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 54 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 54 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 54 +++++… [View More]+++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 85 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------- .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------- .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------- spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 52 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 52 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 46 ++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ----- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 54 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 54 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 54 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 563 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..658b0740 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..85ed3725 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..345033e0 --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..fc926d47 --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..b8abc5b9 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,40 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +82,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..a0ff2d27 --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..8ed2467d --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..76656b1f --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,46 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4a24064c --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..691e1fb1 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..f344ae95 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,54 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0 [View Less]

1 0

[PATCH v2 compass-ci] sparrow/3-code: fix submit command not found
by Liu Yinsi 03 Dec '20

03 Dec '20

[why] when execute submit command in a new machine, error message: [root@localhost 3-code]# submit -bash: submit: command not found because of no '' around EOF, the command cat will parse variable, cause PATH=$PATH:/sbin:/sbin:/bin, but we want is PATH=$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin [how] use 'EOF' to avoid $CCI_SRC, $LKP_SRC to be parsed in cat command. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 … [View More]

1 0

[PATCH v2 compass-ci] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..5dc68c8 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,28 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job … [View More]

1 0

[PATCH compass-ci] sparrow/3-code: fix submit command not found
by Liu Yinsi 03 Dec '20

03 Dec '20

[why] when execute submit command in a new machine, error message: [root@localhost 3-code]# submit -bash: submit: command not found [how] use 'EOF' to avoid $CCI_SRC, $LKP_SRC to be parsed in cat command and cause failed to set ENV $PATH. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 444281e..34cbe17 100755 --- a/sparrow/3-code/dev-env +++ … [View More]

2 2

[PATCH v3 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 03 Dec '20

03 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..17197e7 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,17 @@ +# … [View More]

3 3

[PATCH compass-ci] doc/manual: update submit命令详解.md
by Lin Jiaxin 03 Dec '20

03 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- ...submit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 4 ++++ 1 file changed, 4 insertions(+) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 0edb045..1b6ce8a 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.… [View More]

2 1

[PATCH v4 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it needs to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 57 +++++… [View More]+++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 88 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 +++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 596 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..ba732182 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..4b6bbacc --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..9476bf3c --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..e1f2bbf6 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0 [View Less]

2 2

[PATCH compass-ci 1/2] sbin/mail-job: add help option
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] It's more friendly to user that command-line tool have a help option Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/mail-job | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/sbin/mail-job b/sbin/mail-job index 954c149..85e37a1 100755 --- a/sbin/mail-job +++ b/sbin/mail-job @@ -3,11 +3,32 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Usage: -# mail-job … [View More]

3 5

[PATCH v2 compass-ci] lib/mail_job_result.rb: fix NoMethodError
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] /home/lwt/compass-ci/lib/mail_job_result.rb:45:in `query_job': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 29a9c1a..50e2f53 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -33,7 +33,7 @@ class MailJobResult def set_submitter_info job = … [View More]

1 1

[PATCH compass-ci] scheduler: fix the kernel_params values displayed in job.yaml
by Bai Jing 03 Dec '20

03 Dec '20

[original values] kernel_params: ' user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro rdinit=/sbin/init prompt_ramdisk=0 initrd=20201022.0.cgz initrd=run-ipconfig_20200904.cgz initrd=modules-4.19.90-2003.cgz initrd=headers-4.19.90-2003.cgz initrd=mpstat_20201104.cgz initrd=spinlock_20201104.cgz initrd=git-1.cgz initrd=1.0-1.cgz initrd=lkp-aarch64.cgz initrd=job.cgz ' [now values] kernel_params: - user=lkp - job=/lkp/scheduled/job.yaml - RESULT_ROOT=/result/… [View More]

1 0

[PATCH v5 compass-ci] 优化部分文档
by Li Ping 03 Dec '20

03 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.en.md | 6 +++--- README.md => README.zh.md | 4 ++-- ...\263\350\257\267\346\265\213\350\257\225\346\234\272.md" | 2 -- ...\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 6 +++--- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 5 files changed, 9 insertions(+), 11 deletions(-) rename README.md => README.zh.md (… [View More]

1 0

[PATCH v4 compass-ci] 优化部分文档
by Li Ping 03 Dec '20

03 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.en.md | 6 +++--- README.md | 4 ++-- ...\263\350\257\267\346\265\213\350\257\225\346\234\272.md" | 2 -- ...\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 6 +++--- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 5 files changed, 9 insertions(+), 11 deletions(-) diff --git a/README.en.md b/README.en.md … [View More]

2 2