February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 20 Feb '21

20 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH compass-ci] auto-submit: use the lib script to config the account
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/auto_submit | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 6d29543..b408158 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,8 +7,7 @@ require 'bunny' require 'json' require 'yaml' require 'set' -require_relative '../lib/build_my_info_client' -require "#{ENV['CCI_SRC']}/container/defconfig" +require_relative '../lib/config_account' # receive message and auto submit job class AutoSubmit @@ -130,23 +129,7 @@ class AutoSubmit end end -def my_email - names = Set.new %w[auto-submit] - - defaults = relevant_defaults(names) - defaults['auto-submit']['my_email'] -end - -def config_yaml - build_my_info = BuildMyInfo.new(my_email) - - my_info = build_my_info.search_my_info - build_my_info.config_default_yaml(my_info) - build_my_info.config_lab_yaml(my_info) -end - -# fix verification problems when submit jobs by auto-submit service -config_yaml +config_yaml('auto-submit') auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] ner/mail-robot: optimize start
by Luan Shengde 20 Feb '21

20 Feb '21

attach the directory instead of the single file. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 3827e2c..bd2a62d 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -31,7 +31,7 @@ cmd = %w[ ] + env + %W[ -v /c/upstream-repos:/c/upstream-repos:ro -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw - -v /etc/compass-ci/conf/sec_admins.yaml:/etc/compass-ci/conf/sec_admins.yaml:ro + -v /etc/compass-ci:/etc/compass-ci:ro -v #{ENV['CCI_SRC']}:/c/compass-ci:rw -w /c/compass-ci/container/mail-robot mail-robot -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/mail-robot: add login command
by Luan Shengde 20 Feb '21

20 Feb '21

add login command for account VM server. case user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..8e3aa6e 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + logging_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + logging_account_vm = assign_account_vm ? logging_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,7 +30,7 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - + #{logging_account_vm} You can use the following info to submit jobs: notice: -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 20 Feb '21

20 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/parse-apply-account-email.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..3de2fb5 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,9 +57,18 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + next unless user_info.include?('my_email') + + users_info << user_info + end + + return users_info end def extract_commit_url -- 2.23.0

1 0

[PATCH compass-ci] delimter: Use delimiter account submit the bisect job
by Cao Xueliang 20 Feb '21

20 Feb '21

Add the config_account.rb to create the account info in container, it will use the specified account when submit the job in container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 1 + lib/config_account.rb | 22 ++++++++++++++++++++++ src/delimiter.rb | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 lib/config_account.rb diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..3cff23a 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -31,6 +31,7 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -w #{DEFAULT_CCI}/src delimiter ] diff --git a/lib/config_account.rb b/lib/config_account.rb new file mode 100644 index 0000000..dc0067c --- /dev/null +++ b/lib/config_account.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "./build_my_info_client" +require_relative "../container/defconfig" + +def my_email(account) + names = %W(#{account}) + + defaults = relevant_defaults(names) + defaults[account]['my_email'] +end + +def config_yaml(account) + build_my_info = BuildMyInfo.new(my_email(account)) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..1ea5de5 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -3,8 +3,10 @@ # frozen_string_literal: true require_relative './delimiter/delimiter' +require_relative '../lib/config_account' begin + config_yaml('delimiter') delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix undefined method `/'
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index d43785b..104f098 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -47,7 +47,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, We found some error/warning(s) and the first bad commit in the below project: - git url: #{@upstream_url/@commit_id} + git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") All error/warning(s) (new ones prefixed by >>): -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow: separate service config
by Cao Xueliang 20 Feb '21

20 Feb '21

Extract service config from defaults/yaml to the service/yaml. Read service/yaml in defconfig.rb and defconfig.sh. Add scheduler/yaml for scheduler set the content to the job. Adjust the order of service config. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 1 + sparrow/3-code/dev-env | 80 +++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 25 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 602b498..b121de8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/service/*.yaml', '/etc/compass-ci/accounts/*.yaml', '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| diff --git a/container/defconfig.sh b/container/defconfig.sh index 7c575ef..c33a790 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -10,6 +10,7 @@ load_cci_defaults() yaml_file=( /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/service/*.yaml /etc/compass-ci/accounts/*.yaml "$HOME"/.config/compass-ci/defaults/*.yaml ) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..cc94e01 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -11,52 +11,82 @@ server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -: ${os_http_host:=$server_ip} -: ${initrd_http_host:=$server_ip} -: ${initrd_http_port:=8800} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} +: ${OS_HTTP_HOST:=$server_ip} +: ${SEND_MAIL_HOST:=$server_ip} +: ${SEND_MAIL_PORT:=49000} +: ${SRV_HTTP_HOST:=$server_ip} +: ${SRV_HTTP_PORT:=11300} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} +: ${INITRD_HTTP_HOST:=$server_ip} +: ${INITRD_HTTP_PORT:=8800} +: ${ASSIST_RESULT_HOST:=$server_ip} +: ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} -: ${LOGGING_ES_HOST:=$server_ip} -: ${LOGGING_ES_PORT:=9202} -: ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_PORT:=3080} -: ${ES_HOST:=$server_ip} -: ${ES_PORT:=9200} -: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +lab: nolab +EOF + +mkdir -p /etc/compass-ci/service +cat > /etc/compass-ci/service/service-env.yaml <<EOF +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER -TASKQUEUE_HOST: $sched_host -OS_HTTP_HOST: $os_http_host -INITRD_HTTP_HOST: $initrd_http_host -INITRD_HTTP_PORT: $initrd_http_port +OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port +SRV_HTTP_HOST: $SRV_HTTP_HOST +SRV_HTTP_PORT: $SRV_HTTP_PORT +TASKQUEUE_HOST: $sched_host +SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_PORT: $SEND_MAIL_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT +INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_PORT: $INITRD_HTTP_PORT +ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT -LOGGING_ES_HOST: $LOGGING_ES_HOST -LOGGING_ES_PORT: $LOGGING_ES_PORT -SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT -LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci -ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT -lab: nolab -ES_HOST: $ES_HOST -ES_PORT: $ES_PORT -SRV_HTTP_PORT: $SRV_HTTP_PORT +LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +EOF + +mkdir -p /etc/compass-ci/scheduler +cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF +LKP_SERVER: +GIT_SERVER: +GITCACHE_HOST: +GITCACHE_PORT: +SRV_HTTP_HOST: +SRV_HTTP_PORT: +SEND_MAIL_HOST: +SEND_MAIL_PORT: +INITRD_HTTP_HOST: +INITRD_HTTP_PORT: +ASSIST_RESULT_HOST: +ASSIST_RESULT_PORT: +RESULT_WEBDAV_HOST: +RESULT_WEBDAV_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 20 Feb '21

20 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index 1afc22f..c3eeaf4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..bf8cd8a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def get_service_env + hash = Hash(String, JSON::Any).new + yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + return Hash(String, JSON::Any).from_json(yaml_any.to_json) + end + + def testbox_env(flag = "local") + service_env = get_service_env + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/scheduler/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + hash.each do |key, value| + if value == nil + hash[key] = service_env[key] + end + end + + hash + end +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: modify the function get_hosts_file
by Li Ping 20 Feb '21

20 Feb '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [reason]: get_hosts_file use tbox_group to search for hosts_file --- lib/job.rb | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..8f8c91b28 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_host = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts" + lkp_host = "#{LKP_SRC}/hosts" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + check_files = ["#{lab_host}/#{@job['testbox']}", + "#{lkp_host}/#{@job['testbox']}", + "#{lab_host}/#{tbox_group}", + "#{lkp_host}/#{tbox_group}"] + + check_files.each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 2