November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v8 lkp-tests 2/2] tests/iozone: change the way of parsing parameter
by Lu Kaiyi 10 Nov '20

10 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] use case ... esac for flow control to make parsing more efficient. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 lkp-tests] lib/monitor: query key support regular match
by Wu Zhende 10 Nov '20

10 Nov '20

Enables more flexible monitoring conditions. When I use "monitor job.*=", will get "query=>{"job":{"*":null}}". It's not what I want. I want "query=>{"job.*": null}". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 56283e10..67c2389a 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -51,7 +51,7 @@ class Monitor def merge_overrides return if @overrides.empty? - revise_hash(@query, @overrides, true) + @query.merge!(@overrides) end def field_check -- 2.23.0

2 1

[PATCH compass-ci 3/3] [multi-docker] obtain the queues parameter from CLI
by Xiao Shenwei 10 Nov '20

10 Nov '20

Usage: multi-docker -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how much container do you need -q, --queues queues separated by "," -h, --help show this message example: ./multi-docker -n dc-1g.taishan200-2280-2s48p-256g--a1 \ -c 1 \ -q dc-1g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-docker | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/providers/multi-docker b/providers/multi-docker index 64cafaa..54db8ab 100755 --- a/providers/multi-docker +++ b/providers/multi-docker @@ -9,10 +9,10 @@ require_relative './docker/docker' opt = {} options = OptionParser.new do |opts| - opts.banner = 'Usage: multi-qemu [-t] [-n]' + opts.banner = 'Usage: multi-docker -n -c -q' opts.separator '' - opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| opt['hostname_prefix'] = name end @@ -20,6 +20,10 @@ options = OptionParser.new do |opts| opt['nr_container'] = num end + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -28,13 +32,14 @@ end if ARGV.size.zero? puts options - exit + exit 1 end options.parse!(ARGV) -hostname = opt['hostname_prefix'] || 'dc-1g' +hostname = opt['hostname_prefix'] || "dc-1g.#{ENV['HOSTNAME']}" nr_container = opt['nr_container'] || 1 +queues = opt['queues'] || "dc-1g.#{ENV['HOSTNAME']}" -pids = multi_docker(hostname, nr_container) +pids = multi_docker(hostname, nr_container, queues) save_pid pids -- 2.23.0

2 1

[PATCH compass-ci] change the format of hostname
by Xiao Shenwei 10 Nov '20

10 Nov '20

hostname: before: dc-1g--taishan200-2280-2s48p-256g--a1 now : dc-1g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 7234dfa..9edad62 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -13,7 +13,7 @@ if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then else tbox_group=$hostname fi -host=${tbox_group%%--*} +host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 10 Nov '20

10 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 35 ++++++++++++++++++++++++++++++- src/lib/web_backend.rb | 15 +++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..fb519de 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -31,8 +33,39 @@ get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 # upstream_repo: +# - regex search +# > optional +# +# os: +# > optional +# +# os_version: +# > optional +# +# os_arch: +# > optional +# +# suite: +# > optional +# +# category: +# > optional +# +# testbox: +# > optional +# +# job_state: +# > optional +# +# id: +# > optional +# +# start_date: +# > optional +# +# end_date: # > optional # # page_size: diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

2 3

[PATCH v9 compass-ci 3/3] assign-account: add my_info for request data
by Luan Shengde 10 Nov '20

10 Nov '20

add my_email, my_name, my_uuid in assign-account request data [why] my_email, my_name, my_uuid is required to config the default yaml file ~/.config/compass-ci/defaults/account.yaml [how] add my_email, my_name, my_uuid in assign-account request data when sending assign-account request. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 94 +++++++++++--------- 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..686a327 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,21 +76,24 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: Thank you for joining us. You can use the following command to login the jumper server: - login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} - account password: - #{acct_infos['passwd']} + Suggest: + If you use the password to login, change it in time. regards compass-ci @@ -94,26 +102,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if my_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key my_commit_url is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v9 compass-ci 2/3] assign-account: add config default yaml
by Luan Shengde 10 Nov '20

10 Nov '20

when assigning account, add config the default yaml file [why] easier for user to config the default yaml file [how] write my_email, my_name, my_uuid to the default yaml file ~/.config/compass-ci/default/account.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 80 ++++++++++++-------- 1 file changed, 49 insertions(+), 31 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..137f5c6 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -�� assigned-users -�� user1 -�� user2 -�� user3 -�� ... -�� available-users -�� user11 -�� user12 -�� user13 -�� ... -�� jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -30,9 +30,10 @@ API: call graph: setup_jumper_account_info read_account_info - build_account_name + build_account_info read_jumper_info - setup_authorized_key + config_default_yaml + config_authorized_key the returned data for setup_jumper_account_info like: { @@ -44,6 +45,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +96,50 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def config_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + # my_email, my_name, my_uuid is required to config default yaml file + # they are added along with 'my_ssh_pubkey' when sending assign account request + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 10 Nov '20

10 Nov '20

check my_info for the request data [why] my_info is required when assigning account. my_info can be used to config the default yaml file [how] add check my_info before handle the request my_info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..c0753b0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + assign_jumper_account(data) +end + +def assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

1 0

[PATCH compass-ci] sched: refactor sched class for the submit job
by Ren Wen 10 Nov '20

10 Nov '20

According to scheduler.cr API to refactor sched class. Extract submit_job function from sched.cr to submit_job.cr Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 172 +---------------------------------- src/scheduler/submit_job.cr | 175 ++++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+), 171 deletions(-) create mode 100644 src/scheduler/submit_job.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 1b6eb8a..ba89ab9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -13,6 +13,7 @@ require "../scheduler/jobfile_operate" require "../scheduler/redis_client" require "../scheduler/elasticsearch_client" +require "../scheduler/submit_job" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" require "../scheduler/close_job" @@ -51,177 +52,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned - def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) - lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) - end - - def get_commit_date(job) - if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") - data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", - "git_command": ["git-log", "--pretty=format:%cd", "--date=unix", - "#{job["upstream_commit"]}", "-1"]})) - response = @rgc.git_command(data) - return response.body if response.status_code == 200 - end - - return nil - end - - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end - - job_content = JSON.parse(body) - job = Job.new(job_content, job_content["id"]?) - job["commit_date"] = get_commit_date(job) - - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' - cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") - return submit_single_job(job) - else - cluster_config = get_cluster_config(cluster_file, - job.lkp_initrd_user, - job.os_arch) - return submit_cluster_job(job, cluster_config) - end - rescue ex - puts ex.inspect_with_backtrace - return [{ - "job_id" => "0", - "message" => ex.to_s, - "job_state" => "submit", - }] - end - - # return: - # success: [{"job_id" => job_id1, "message => "", "job_state" => "submit"}, ...] - # failure: [..., {"job_id" => 0, "message" => err_msg, "job_state" => "submit"}] - def submit_cluster_job(job, cluster_config) - job_messages = Array(Hash(String, String)).new - lab = job.lab - - # collect all job ids - job_ids = [] of String - - net_id = "192.168.222" - ip0 = cluster_config["ip0"]? - if ip0 - ip0 = ip0.as_i - else - ip0 = 1 - end - - # steps for each host - cluster_config["nodes"].as_h.each do |host, config| - tbox_group = host.to_s - job_id = add_task(tbox_group, lab) - - # return when job_id is '0' - # 2 Questions: - # - how to deal with the jobs added to DB prior to this loop - # - may consume job before all jobs done - return job_messages << { - "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", - "job_state" => "submit", - } unless job_id - - job_ids << job_id - - # add to job content when multi-test - job["testbox"] = tbox_group - job.update_tbox_group(tbox_group) - job["node_roles"] = config["roles"].as_a.join(" ") - direct_macs = config["macs"].as_a - direct_ips = [] of String - direct_macs.size.times do - raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 - direct_ips << "#{net_id}.#{ip0}" - ip0 += 1 - end - job["direct_macs"] = direct_macs.join(" ") - job["direct_ips"] = direct_ips.join(" ") - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - job_messages << { - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", - "result_root" => "/srv#{job.result_root}", - } - return job_messages if response["error"]? - end - - cluster_id = job_ids[0] - - # collect all host states - cluster_state = Hash(String, Hash(String, String)).new - job_ids.each do |job_id| - cluster_state[job_id] = {"state" => ""} - # will get cluster id according to job id - @redis.hash_set("sched/id2cluster", job_id, cluster_id) - end - - @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) - - return job_messages - end - - # return: - # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] - # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] - def submit_single_job(job) - queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue - - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - - job_id = add_task(queue, job.lab) - return [{ - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", - "job_state" => "submit", - }] unless job_id - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - - return [{ - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", - "result_root" => "/srv#{job.result_root}", - }] - end - - # return job_id - def add_task(queue, lab) - task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) - response = @task_queue.add_task("sched/#{queue}", task_desc) - JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 - end - - # add job content to es and return a response - def add_job(job, job_id) - job.update_id(job_id) - @es.set_job_content(job) - end - def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" extra_job_fields = [ diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr new file mode 100644 index 0000000..8871464 --- /dev/null +++ b/src/scheduler/submit_job.cr @@ -0,0 +1,175 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def submit_job(env : HTTP::Server::Context) + body = env.request.body.not_nil!.gets_to_end + + job_content = JSON.parse(body) + job = Job.new(job_content, job_content["id"]?) + job["commit_date"] = get_commit_date(job) + + # it is not a cluster job if cluster field is empty or + # field's prefix is 'cs-localhost' + cluster_file = job["cluster"] + if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + return submit_single_job(job) + else + cluster_config = get_cluster_config(cluster_file, + job.lkp_initrd_user, + job.os_arch) + return submit_cluster_job(job, cluster_config) + end + rescue ex + puts ex.inspect_with_backtrace + return [{ + "job_id" => "0", + "message" => ex.to_s, + "job_state" => "submit", + }] + end + + # return: + # success: [{"job_id" => job_id1, "message => "", "job_state" => "submit"}, ...] + # failure: [..., {"job_id" => 0, "message" => err_msg, "job_state" => "submit"}] + def submit_cluster_job(job, cluster_config) + job_messages = Array(Hash(String, String)).new + lab = job.lab + + # collect all job ids + job_ids = [] of String + + net_id = "192.168.222" + ip0 = cluster_config["ip0"]? + if ip0 + ip0 = ip0.as_i + else + ip0 = 1 + end + + # steps for each host + cluster_config["nodes"].as_h.each do |host, config| + tbox_group = host.to_s + job_id = add_task(tbox_group, lab) + + # return when job_id is '0' + # 2 Questions: + # - how to deal with the jobs added to DB prior to this loop + # - may consume job before all jobs done + return job_messages << { + "job_id" => "0", + "message" => "add task queue sched/#{tbox_group} failed", + "job_state" => "submit", + } unless job_id + + job_ids << job_id + + # add to job content when multi-test + job["testbox"] = tbox_group + job.update_tbox_group(tbox_group) + job["node_roles"] = config["roles"].as_a.join(" ") + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 + direct_ips << "#{net_id}.#{ip0}" + ip0 += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") + + response = add_job(job, job_id) + message = (response["error"]? ? response["error"]["root_cause"] : "") + job_messages << { + "job_id" => job_id, + "message" => message.to_s, + "job_state" => "submit", + "result_root" => "/srv#{job.result_root}", + } + return job_messages if response["error"]? + end + + cluster_id = job_ids[0] + + # collect all host states + cluster_state = Hash(String, Hash(String, String)).new + job_ids.each do |job_id| + cluster_state[job_id] = {"state" => ""} + # will get cluster id according to job id + @redis.hash_set("sched/id2cluster", job_id, cluster_id) + end + + @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) + + return job_messages + end + + # return: + # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] + # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] + def submit_single_job(job) + queue = job.queue + return [{ + "job_id" => "0", + "message" => "get queue failed", + "job_state" => "submit", + }] unless queue + + # only single job will has "idle job" and "execute rate limiter" + if job["idle_job"].empty? + queue += "#{job.get_uuid_tag}" + else + queue = "#{queue}/idle" + end + + job_id = add_task(queue, job.lab) + return [{ + "job_id" => "0", + "message" => "add task queue sched/#{queue} failed", + "job_state" => "submit", + }] unless job_id + + response = add_job(job, job_id) + message = (response["error"]? ? response["error"]["root_cause"] : "") + + return [{ + "job_id" => job_id, + "message" => message.to_s, + "job_state" => "submit", + "result_root" => "/srv#{job.result_root}", + }] + end + + # return job_id + def add_task(queue, lab) + task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) + response = @task_queue.add_task("sched/#{queue}", task_desc) + JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 + end + + # add job content to es and return a response + def add_job(job, job_id) + job.update_id(job_id) + @es.set_job_content(job) + end + + # get cluster config using own lkp_src cluster file, + # a hash type will be returned + def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) + lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + return YAML.parse(File.read(cluster_file_path)) + end + + def get_commit_date(job) + if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") + data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", + "git_command": ["git-log", "--pretty=format:%cd", "--date=unix", + "#{job["upstream_commit"]}", "-1"]})) + response = @rgc.git_command(data) + return response.body if response.status_code == 200 + end + + return nil + end +end -- 2.23.0

1 0

[PATCH compass-ci] sched: refactor sched class for the update job parameter
by Cao Xueliang 10 Nov '20

10 Nov '20

According to scheduler.cr API to refactor sched class. Extract update_job_parameter function from sched.cr to update_job_parameter.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 31 +----------------------- src/scheduler/update_job_parameter.cr | 34 +++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 30 deletions(-) create mode 100644 src/scheduler/update_job_parameter.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 1b6eb8a..e966ae2 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -17,6 +17,7 @@ require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" require "../scheduler/close_job" require "../scheduler/request_cluster_state" +require "../scheduler/update_job_parameter" class Sched property es @@ -236,36 +237,6 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? - if !job_id - return false - end - - # try to get report value and then update it - job_content = {} of String => String - job_content["id"] = job_id - - (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? - if !value || value == "" - next - end - if parameter == "start_time" || parameter == "end_time" - value = Time.unix(value.to_i).to_local.to_s("%Y-%m-%d %H:%M:%S") - end - - job_content[parameter] = value - end - - @redis.update_job(job_content) - - # json log - log = job_content.dup - log["job_id"] = log.delete("id").not_nil! - return log.to_json - end - def update_tbox_wtmp(env : HTTP::Server::Context) testbox = "" hash = Hash(String, String).new diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr new file mode 100644 index 0000000..0692025 --- /dev/null +++ b/src/scheduler/update_job_parameter.cr @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def update_job_parameter(env : HTTP::Server::Context) + job_id = env.params.query["job_id"]? + if !job_id + return false + end + + # try to get report value and then update it + job_content = {} of String => String + job_content["id"] = job_id + + (%w(start_time end_time loadavg job_state)).each do |parameter| + value = env.params.query[parameter]? + if !value || value == "" + next + end + if parameter == "start_time" || parameter == "end_time" + value = Time.unix(value.to_i).to_local.to_s("%Y-%m-%d %H:%M:%S") + end + + job_content[parameter] = value + end + + @redis.update_job(job_content) + + # json log + log = job_content.dup + log["job_id"] = log.delete("id").not_nil! + return log.to_json + end +end -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci November 2020