- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sbin/build-my-info: fix no key: my_ssh_pubkey
by Luan Shengde 20 Feb '21

20 Feb '21

[why] it will read my_ssh_pubkey and check it when submit jobs. $ submit -m -c job.yaml -i ~/lkp-tests/jobs/ssh-on-fail.yaml submit_id=f87d2951-c226-4e51-a3e4-c71818a4c974 submit job.yaml failed, got job id=0, error: Missing hash key: "my_ssh_pubkey" [how] add initialize my_ssh_pubkey when building my_info. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index c7a7798..ecfcb42 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,6 +12,7 @@ def build_my_info(option) info_es = build_my_info.search_my_info info_es.update option info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + info_es['my_ssh_pubkey'] = [] build_my_info.config_my_info(info_es) end -- 2.23.0

1 0

[PATCH v2 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 20 Feb '21

20 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH compass-ci] auto-submit: use the lib script to config the account
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/auto_submit | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 6d29543..b408158 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,8 +7,7 @@ require 'bunny' require 'json' require 'yaml' require 'set' -require_relative '../lib/build_my_info_client' -require "#{ENV['CCI_SRC']}/container/defconfig" +require_relative '../lib/config_account' # receive message and auto submit job class AutoSubmit @@ -130,23 +129,7 @@ class AutoSubmit end end -def my_email - names = Set.new %w[auto-submit] - - defaults = relevant_defaults(names) - defaults['auto-submit']['my_email'] -end - -def config_yaml - build_my_info = BuildMyInfo.new(my_email) - - my_info = build_my_info.search_my_info - build_my_info.config_default_yaml(my_info) - build_my_info.config_lab_yaml(my_info) -end - -# fix verification problems when submit jobs by auto-submit service -config_yaml +config_yaml('auto-submit') auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] ner/mail-robot: optimize start
by Luan Shengde 20 Feb '21

20 Feb '21

attach the directory instead of the single file. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 3827e2c..bd2a62d 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -31,7 +31,7 @@ cmd = %w[ ] + env + %W[ -v /c/upstream-repos:/c/upstream-repos:ro -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw - -v /etc/compass-ci/conf/sec_admins.yaml:/etc/compass-ci/conf/sec_admins.yaml:ro + -v /etc/compass-ci:/etc/compass-ci:ro -v #{ENV['CCI_SRC']}:/c/compass-ci:rw -w /c/compass-ci/container/mail-robot mail-robot -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/mail-robot: add login command
by Luan Shengde 20 Feb '21

20 Feb '21

add login command for account VM server. case user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..8e3aa6e 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + logging_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + logging_account_vm = assign_account_vm ? logging_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,7 +30,7 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - + #{logging_account_vm} You can use the following info to submit jobs: notice: -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 20 Feb '21

20 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/parse-apply-account-email.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..3de2fb5 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,9 +57,18 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + next unless user_info.include?('my_email') + + users_info << user_info + end + + return users_info end def extract_commit_url -- 2.23.0

1 0

[PATCH compass-ci] delimter: Use delimiter account submit the bisect job
by Cao Xueliang 20 Feb '21

20 Feb '21

Add the config_account.rb to create the account info in container, it will use the specified account when submit the job in container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 1 + lib/config_account.rb | 22 ++++++++++++++++++++++ src/delimiter.rb | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 lib/config_account.rb diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..3cff23a 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -31,6 +31,7 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -w #{DEFAULT_CCI}/src delimiter ] diff --git a/lib/config_account.rb b/lib/config_account.rb new file mode 100644 index 0000000..dc0067c --- /dev/null +++ b/lib/config_account.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "./build_my_info_client" +require_relative "../container/defconfig" + +def my_email(account) + names = %W(#{account}) + + defaults = relevant_defaults(names) + defaults[account]['my_email'] +end + +def config_yaml(account) + build_my_info = BuildMyInfo.new(my_email(account)) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..1ea5de5 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -3,8 +3,10 @@ # frozen_string_literal: true require_relative './delimiter/delimiter' +require_relative '../lib/config_account' begin + config_yaml('delimiter') delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix undefined method `/'
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index d43785b..104f098 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -47,7 +47,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, We found some error/warning(s) and the first bad commit in the below project: - git url: #{@upstream_url/@commit_id} + git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") All error/warning(s) (new ones prefixed by >>): -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow: separate service config
by Cao Xueliang 20 Feb '21

20 Feb '21

Extract service config from defaults/yaml to the service/yaml. Read service/yaml in defconfig.rb and defconfig.sh. Add scheduler/yaml for scheduler set the content to the job. Adjust the order of service config. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 1 + sparrow/3-code/dev-env | 80 +++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 25 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 602b498..b121de8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/service/*.yaml', '/etc/compass-ci/accounts/*.yaml', '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| diff --git a/container/defconfig.sh b/container/defconfig.sh index 7c575ef..c33a790 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -10,6 +10,7 @@ load_cci_defaults() yaml_file=( /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/service/*.yaml /etc/compass-ci/accounts/*.yaml "$HOME"/.config/compass-ci/defaults/*.yaml ) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..cc94e01 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -11,52 +11,82 @@ server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -: ${os_http_host:=$server_ip} -: ${initrd_http_host:=$server_ip} -: ${initrd_http_port:=8800} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} +: ${OS_HTTP_HOST:=$server_ip} +: ${SEND_MAIL_HOST:=$server_ip} +: ${SEND_MAIL_PORT:=49000} +: ${SRV_HTTP_HOST:=$server_ip} +: ${SRV_HTTP_PORT:=11300} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} +: ${INITRD_HTTP_HOST:=$server_ip} +: ${INITRD_HTTP_PORT:=8800} +: ${ASSIST_RESULT_HOST:=$server_ip} +: ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} -: ${LOGGING_ES_HOST:=$server_ip} -: ${LOGGING_ES_PORT:=9202} -: ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_PORT:=3080} -: ${ES_HOST:=$server_ip} -: ${ES_PORT:=9200} -: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +lab: nolab +EOF + +mkdir -p /etc/compass-ci/service +cat > /etc/compass-ci/service/service-env.yaml <<EOF +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER -TASKQUEUE_HOST: $sched_host -OS_HTTP_HOST: $os_http_host -INITRD_HTTP_HOST: $initrd_http_host -INITRD_HTTP_PORT: $initrd_http_port +OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port +SRV_HTTP_HOST: $SRV_HTTP_HOST +SRV_HTTP_PORT: $SRV_HTTP_PORT +TASKQUEUE_HOST: $sched_host +SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_PORT: $SEND_MAIL_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT +INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_PORT: $INITRD_HTTP_PORT +ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT -LOGGING_ES_HOST: $LOGGING_ES_HOST -LOGGING_ES_PORT: $LOGGING_ES_PORT -SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT -LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci -ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT -lab: nolab -ES_HOST: $ES_HOST -ES_PORT: $ES_PORT -SRV_HTTP_PORT: $SRV_HTTP_PORT +LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +EOF + +mkdir -p /etc/compass-ci/scheduler +cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF +LKP_SERVER: +GIT_SERVER: +GITCACHE_HOST: +GITCACHE_PORT: +SRV_HTTP_HOST: +SRV_HTTP_PORT: +SEND_MAIL_HOST: +SEND_MAIL_PORT: +INITRD_HTTP_HOST: +INITRD_HTTP_PORT: +ASSIST_RESULT_HOST: +ASSIST_RESULT_PORT: +RESULT_WEBDAV_HOST: +RESULT_WEBDAV_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 20 Feb '21

20 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index 1afc22f..c3eeaf4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..bf8cd8a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def get_service_env + hash = Hash(String, JSON::Any).new + yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + return Hash(String, JSON::Any).from_json(yaml_any.to_json) + end + + def testbox_env(flag = "local") + service_env = get_service_env + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/scheduler/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + hash.each do |key, value| + if value == nil + hash[key] = service_env[key] + end + end + + hash + end +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: modify the function get_hosts_file
by Li Ping 20 Feb '21

20 Feb '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [reason]: get_hosts_file use tbox_group to search for hosts_file --- lib/job.rb | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..8f8c91b28 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_host = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts" + lkp_host = "#{LKP_SRC}/hosts" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + check_files = ["#{lab_host}/#{@job['testbox']}", + "#{lkp_host}/#{@job['testbox']}", + "#{lab_host}/#{tbox_group}", + "#{lkp_host}/#{tbox_group}"] + + check_files.each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: various cleanup
by Wu Fengguang 20 Feb '21

20 Feb '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- tests/rpmbuild-pkg | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 781d04a0d..9f455f0d0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,8 +5,8 @@ . $LKP_SRC/lib/upload.sh [ -n "$rpm_repo" ] || die "rpm_repo is empty" -benchmark=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" +package_name=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { @@ -16,15 +16,15 @@ init_workspace() get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" } download_rpm_repo() { - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in ${filelist[*]} + for pkgfile in "${filelist[@]}" do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" @@ -34,20 +34,29 @@ download_rpm_repo() build_rpm() { - spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` + local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec + + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + # Install build depends - yum-builddep -y $spec_dir + yum-builddep -y $spec_file || exit # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_dir + spectool -g -R $spec_file || exit # Building rpm or srpm packages - rpmbuild -ba $spec_dir + rpmbuild -ba $spec_file || exit +} + +show_rpm_files() +{ + find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" + find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" - for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + local file + for file in $(show_rpm_files) do upload_one_curl ${file} ${sync_dest} done -- 2.23.0

1 0

[PATCH v2 compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..5a9bd2ddadec 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/install/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..334f5c2c4a69 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: add a test ping condition to network_ok
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] In physical machine, there will be a delay between the start of network card and the availability of the network. So we add a test ping condition to network_ok. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 06aa09bf65b2..38965ff60755 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -85,14 +85,19 @@ net_devices_link() done } +test_ping() +{ + ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 +} + network_ok() { local i for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && test_ping && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done is_clearlinux && { -- 2.23.0

2 7

[PATCH compass-ci] lib/git_mirror.rb: change the way to load DEFAULTS file
by Li Yuanchao 19 Feb '21

19 Feb '21

Before: DEFAULTS files are individual. After: Load DEFAULTS file and merge it with the parent directory DEFAULTS files. So the outer DEFAULTS files would keep common things for the inner ones. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 39b1619..346352b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -130,6 +130,7 @@ class MirrorMain defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) end def traverse_repodir(repodir) @@ -224,7 +225,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo) + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -449,15 +450,17 @@ end # main thread class MirrorMain - def merge_defaults(git_repo) - defaults_key = File.dirname(git_repo) + def merge_defaults(object_key, object) + return object if object_key == 'default' + + defaults_key = File.dirname(object_key) while defaults_key != '.' - return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(object) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + return @defaults['default'].merge(object) if @defaults['default'] - return @git_info[git_repo] + return object end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] lkp-bootstrap: delete unnecessary exit
by Xu Xijian 19 Feb '21

19 Feb '21

The exit cmd after reboot is unnecessary, delete it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- rootfs/addon/etc/init.d/lkp-bootstrap | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/addon/etc/init.d/lkp-bootstrap b/rootfs/addon/etc/init.d/lkp-bootstrap index f66ec4eb5..091b29890 100755 --- a/rootfs/addon/etc/init.d/lkp-bootstrap +++ b/rootfs/addon/etc/init.d/lkp-bootstrap @@ -24,7 +24,6 @@ if [ -s /lkp/run/lkp-bootstrap.pid ]; then # the previous lkp-bootstrap init.d script failed to restart the machine echo "LKP bootstrap: first run may failed, retry rebooting" reboot - exit fi read_kernel_cmdline_vars() -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 19 Feb '21

19 Feb '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] src/scheduler/kernel_params.cr: add noserverino option to cifs mount
by Xu Xijian 19 Feb '21

19 Feb '21

[Log] [ 7.507105] CIFS VFS: Autodisabling the use of server inode numbers on \\172.168.131.113\os. This server doesn't seem to support them properly. Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 6fbe405..bb5983f 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,7 +18,7 @@ class Job fs2root = { "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] doc/support-os-os_arch-os_version.md: add 20.03-SP1
by Xu Xijian 19 Feb '21

19 Feb '21

1. add 20.03-SP1 version to this doc, it's already supported 2. adjust some lines to be aligned Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..f91e91b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,24 @@ ``` os os_arch os_version supported-os_mount openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, ,initramfs - 20.09 cifs/nfs, ,initramfs + 20.03 cifs/nfs, ,initramfs + 20.03-SP1 cifs/nfs, + 20.09 cifs/nfs, ,initramfs mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, + 20.03-SP1 cifs/nfs, 20.09 cifs/nfs, centos aarch64 7 container, - 7.6.1810 cifs/nfs, ,initramfs - 7.8.2003 cifs/nfs, ,initramfs + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, ,initramfs +debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH lkp-tests] jobs/rpmbuild-pkg.yaml: change the rpm_repo field to level-1 parameter
by Li Ping 19 Feb '21

19 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 19fd5ec28..53e145b3b 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional +rpm_repo: rpmbuild-pkg: - rpm_repo: -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: fix no such file error
by Li Ping 19 Feb '21

19 Feb '21

[why]: grep: 2ping.spec: No such file or directory sed: no input files --- tests/rpmbuild-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 156ca0f63..781d04a0d 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -35,7 +35,7 @@ download_rpm_repo() build_rpm() { spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` # Install build depends yum-builddep -y $spec_dir # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 94 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..226251a8bace --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,94 @@ +# Summary +========= + +`os_mount` defines the mount method of testbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + - optional {kernel_append_root}: + - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. + and you can use the root partition data of this job in the futher job by {use_root_partition}. + - use_root_partition : give an iconic string, then the root partition data generated by the + pervious job which has the {save_root_partition} field will be used. + - root_partition_size: specify the size of root partition, *default is 10G*. + + - tips: + - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. + - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + +2. scheduler return the ipxe_str to testbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. dracut step of boot + ``` + if have {use_root_partition}; then + base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 + + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + + if {save_root_partition} == {use_root_partition}; then + boot from /dev/mappaer/os-{boot_lv_name} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + else + boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} + boot from /dev/mappaer/os-{boot_lv_name} + fi + + else: + base_lv_name=openeuler_aarch64_20.03_{timestamp} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + # create logical volume + lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + + # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly + lvchange -p r /dev/mapper/os-{base_lv_name} + } + + boot_lv_name=openeuler_aarch64_20.03 + if have {save_root_partition}; then + boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 + boot from /dev/mappaer/os-{boot_lv_name} + fi + ``` + +4. execute the job -- 2.23.0

2 2

[PATCH compass-ci] sched: add rpmbuild-pkg upload dir
by Cao Xueliang 19 Feb '21

19 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b724b07..1afc22f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -154,9 +154,9 @@ class Job set_submit_date() set_pp_params() set_rootfs() + set_os_mount() set_result_root() set_result_service() - set_os_mount() set_depends_initrd() set_kernel() set_initrds_uri() @@ -247,7 +247,7 @@ class Job def get_pkg_common_dir pkg_style = nil - ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| pkg_style = @hash[item]? break if pkg_style end @@ -276,6 +276,8 @@ class Job package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" + elsif @hash["rpmbuild-pkg"]? + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: optimize the test_ping logic
by Yu Chuan 19 Feb '21

19 Feb '21

[Why] 1. '-W' => '-w' -w <deadline> reply wait <deadline> in seconds -W <timeout> time to wait for response deadline(-w) is more appropriate than timeout(-W). 2. delete 'return 1' If ping does not receive any reply packets at all it will exit with code 1. If a packet count and deadline are both specified, and fewer than count packets are received by the time the deadline has arrived, it will also exit with code 1. On other error it exits with code 2. Otherwise it exits with code 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 38965ff60755..76f07e6a7669 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,7 +87,7 @@ net_devices_link() test_ping() { - ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 + ping -c 1 -w 1 $LKP_SERVER > /dev/null } network_ok() -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/rpmbuild-pkg: add depend pkg
by Wei Jihui 19 Feb '21

19 Feb '21

[Error] yum-builddep: command not found [How] %yum provides yum-builddep dnf-plugins-core-4.0.11-5.oe1.noarch : Core Plugins for DNF Repo : @System Matched from: Filename : /usr/bin/yum-builddep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/rpmbuild-pkg b/distro/depends/rpmbuild-pkg index 6f00a6a2..50fd9b75 100644 --- a/distro/depends/rpmbuild-pkg +++ b/distro/depends/rpmbuild-pkg @@ -1,2 +1,3 @@ build-essential rpmdevtools +dnf-plugins-core -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci