- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] jobs: add libmicro.yaml
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..446ab23b --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,4 @@ +suite: libmicro +category: benchmark + +libmicro: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix bug when program-0 add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..67bd6d5e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,9 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + if program =~ /^(.*)-\d+?/ + program = $1 + end # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

4 6

[PATCH v5 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

2 3

[PATCH lkp-tests 2/2] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

3 3

[PATCH lkp-tests] jobs: add yaml for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..9a577eb8 --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,5 @@ +suite: libmicro +testcase: libmicro +category: benchmark + +libmicro: -- 2.23.0

4 3

[PATCH v2 lkp-tests] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

2 2

[PATCH v5 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 04 Nov '20

04 Nov '20

1. disable use golbal variable apply_info [why]: rubocop: Style/GlobalVars: Do not introduce global variables. [how]: use local variables instead of global variables 2. fix OptionParser argument: [why]: when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how]: write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add transfer user info when execute apply account command [why]: when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how]: transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 92 ++++++++++++-------- 1 file changed, 55 insertions(+), 37 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..bda627d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,65 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, apply_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,10 +78,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +89,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +101,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

2 2

[PATCH compass-ci] kernel_params.cr: add sched_steal_node_limit=8 when test mysql optimization kernel
by Xu Xijian 04 Nov '20

04 Nov '20

[why] When test the kernel with mysql optimization, the parameter "sched_steal_node_limit=8" is required to be added into kernel params. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 54438b5..546f012 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -31,6 +31,9 @@ class Job private def set_kernel_params self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + if "#{linux_vmlinuz_path}".includes?("mysql") + self["kernel_params"] += " sched_steal_node_limit=8" + end end end -- 2.23.0

3 3

[PATCH compass-ci 3/8] container/mail-robot: apply-account-component.rb
by Luan Shengde 04 Nov '20

04 Nov '20

ParseApplyAccountEmail parse commit url and pub_key check out commit url check commit url exists check base url in upstream-repos check commit available gitee.com clone the repo and check the commit exists non-gitee.com check the feedback of curl command to check the commit url available check out pub_key extract the attachment of the pub_key ApplyAccount apply account with apply_account_info check account SendMail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../mail-robot/apply-account-component.rb | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100755 container/mail-robot/apply-account-component.rb diff --git a/container/mail-robot/apply-account-component.rb b/container/mail-robot/apply-account-component.rb new file mode 100755 index 0000000..98272e7 --- /dev/null +++ b/container/mail-robot/apply-account-component.rb @@ -0,0 +1,232 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_info' => '', + 'error_message' => error_message + } + send_err_mail = SendMail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir ENV['ROBOT_TMP_REPOS_DIR'] + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end + +# apply jumper account for user +class ApplyAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['account'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end + +# build mail data and send mail +class SendMail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @my_email = send_mail_info['my_email'] + @my_name = send_mail_info['my_name'] + @error_message = send_mail_info['error_message'] + @my_info = send_mail_info['my_info'] + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: [compass-ci] Account Ready + + Dear #{@my_name}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_uuid: #{@my_info['my_uuid']} + my_email: #{@my_info['my_email']} + my_name: #{@my_info['my_name']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: apply account failed + + Dear #{@my_name} + + Your application for account failed with following errors + + #{@error_message} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

2 8

[PATCH v2 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 04 Nov '20

04 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

2 1