- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..e3dfd17 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo SCHED_PORT: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 0

[PATCH v2 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..6acfc2e 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo sched_port: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 1

[PATCH compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..03bb061 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,16 @@ run_docker() done } +kill_process() +{ + multi_qemu=`ps -ef |grep multi-qemu |grep "$USER" |awk '{print $2}'` + multi_docker=`ps -ef |grep multi-docker |grep "$USER" |awk '{print $2}'` + kill -9 $multi_qemu + kill -9 $multi_docker +} + run_qemu run_docker wait + +kill_process -- 2.23.0

3 2

[PATCH compass-ci] sbin/result2stats: support merge yaml file
by Lu Weitao 12 Jan '21

12 Jan '21

[Why] user may extract $program result to $program.yaml Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index e603763..8444334 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,19 +39,21 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') + when /\.yaml$/ + File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if monitor == 'stats' # stats.json already created? + next if %w[stats job].include?(monitor) extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] stats/crystal/memory--stat.cr: fix some errors manually
by Lu Kaiyi 12 Jan '21

12 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index da7ac0d86..f45ab5fb1 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -6,14 +6,14 @@ def parse cg = "nocg" - $stdin.each_line do |line| + STDIN.each_line do |line| case line when /^time:/ puts line when /^cgroup: (\S+)/ cg = $1 when /(\S+)\s*(\S+)/ - k, v, = line.split + k, v = line.split puts "#{cg}.#{k}: #{v}" end end -- 2.23.0

1 0

[PATCH lkp-tests 2/3] stats/crystal/memory--stat.cr: do first conversion by ruby2crystal tools
by Lu Kaiyi 12 Jan '21

12 Jan '21

use the ruby2crystal tools do first conversion like below: /c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod memory--stat.cr /c/lkp-tests/tools/ruby2crystal.sh memory--stat.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index 1a39e4075..da7ac0d86 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -1,11 +1,11 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # time: 123456 # cgroup: pmbench.1 # cache 0 def parse - cg = 'nocg' + cg = "nocg" $stdin.each_line do |line| case line when /^time:/ -- 2.23.0

1 0

[PATCH compass-ci] addsoftware: add to clean hidden files in runtime dir
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

2 4

[PATCH v1 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..ea0c259 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,33 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + if @hash.has_key?(item) + pkg_style = item + break + end end + return nil unless pkg_style + + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +287,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

$lab will be used in $CCI_SRC/sbin/build-my-info.rb, so add it. can see commit 66b58a7be3cf371c74946950ebe9dc73efd9a16e Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..69b31cd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -43,6 +43,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +lab: nolab EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH v6 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 12 Jan '21

12 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v6 compass-ci 6/7] lib/mail-post: add email counts/mapping checks
by Luan Shengde 12 Jan '21

12 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v6 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 12 Jan '21

12 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v6 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 12 Jan '21

12 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v6 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 12 Jan '21

12 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v6 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v6 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 12 Jan '21

12 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v2 compass-ci] container/auto-submit: fix submit job failed
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user intall compass-ci at their local server, need to verify when submit job in auto-submit container, so store autosubmit_info to ES, and mount yaml file in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- lib/build_my_info_client.rb | 4 ++-- sbin/build-my-info.rb | 4 ++++ 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..d0d4b77 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,12 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = %x(uuidgen).chomp) @lab = lab || 'nolab' @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..d68932e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -31,3 +31,7 @@ options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info + +autosubmit_info = YAML.load_file('/etc/compass-ci/account/autosubmit.yaml') +build_autosubmit_info = BuildMyInfo.new(autosubmit_info['my_email'], autosubmit_info['my_name'], autosubmit_info['lab'], autosubmit_info['my_token']) +build_autosubmit_info.store_account_info -- 2.23.0

2 2

[PATCH v3 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= export_mac_ip: PUB_NIC=$(ip route get 1.2.3.4 | awk '{print $5; exit}') PUB_IP=$( ip route get 1.2.3.4 | awk '{print $7; exit}') network_ok: [ "$(cat $i/operstate)" = 'up' ]&& return 0 when cat /sys/class/net/eth0/operstate is up, network is ok, then it will not sleep and immediately execute export_ip_mac(), but actually route is not ready, it can't get mac and ip by search route, so use ping can ensure save enough time for route. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..df25e99e0 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER > /dev/null && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

3 4

[PATCH compass-ci] sparrow/3-code: add RESULT_WEBDAV_PORT field
by Xiao Shenwei 12 Jan '21

12 Jan '21

[why] $LKP_SRC/lib/upload.sh will upload some result file by result-webdav container [how] display definition in default profile Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..f2366c2 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -19,6 +19,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} : ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_PORT:=3080} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -43,6 +44,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..1c6eb1a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,10 +249,37 @@ class Job set_upload_dirs() end + def get_pkg_common_dir + common_dir = "" + + if @hash["cci-makepkg"]? + pkg_style = "cci-makepkg" + elsif @hash["cci-depends"]? + pkg_style = "cci-depends" + elsif @hash["build-pkg"]? + pkg_style = "build-pkg" + else + pkg_style = nil + end + + if pkg_style + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end + + return common_dir + end + def get_package_dir if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + common_dir = get_pkg_common_dir end if @hash["cci-makepkg"]? -- 2.23.0

2 2

[PATCH lkp-tests] reset-tbox to reboot machine testbox
by Zhang Dewan 11 Jan '21

11 Jan '21

[why] sumbit a job to reboot machine testbox is better. [how] cp ~/lkp-test/jobs/reset-tbox-yaml . submit reset-tbox.yaml host_name="full-name-of-machine-testbox" Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 17 +++++++++++++++++ tests/reset-tbox.yaml | 12 ++++++++++++ 3 files changed, 39 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox create mode 100644 tests/reset-tbox.yaml diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..9c3f4f01 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +testcase: reset-tbox + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset + +host_name: +reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..b0f645d7 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,17 @@ +#!/bin/bash +# host_name + +source "/etc/lab/.account.info" +hosts_file=/etc/lab/hosts_file + +# require $host_name +host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + +if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then + echo ">>> exec ipmitool commamd!" + ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + [ $? -eq 0 ] && echo "succeed!" +else + echo "specify host cannot found0!" + exit +fi diff --git a/tests/reset-tbox.yaml b/tests/reset-tbox.yaml new file mode 100644 index 00000000..f557085f --- /dev/null +++ b/tests/reset-tbox.yaml @@ -0,0 +1,12 @@ +short_description: > + To reboot machine testbox like taishan200-2280-2s64p-256g--aN etc. + +description: | + reset-tbox executed by a special docker which running on ibmc-mgnt, where you can access all server's iBMC-os by ipmitool, and then reboot the host. + +homepage: + +parameters: + host_name: see it on the first line about 'latbox aN' + +results: -- 2.23.0

3 4

[PATCH v4 compass-ci 3/4] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this to adapt the changes to job fields: - 'kernel_uri' - 'kernel_params' - 'initrds_uri' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..5e6044a 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,9 +158,13 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " - response += Array(String).from_json(job.kernel_params).join(" ") + + _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } + response += _initrds_uri.join("\n") + "\n" + + _kernel_params = ["kernel #{job.kernel_uri}"] + Array(String).from_json(job.kernel_params) + Array(String).from_json(job.ipxe_kernel_params) + response += _kernel_params.join(" ") + response += "\nboot\n" return response -- 2.23.0

1 1

[PATCH v4 compass-ci 2/4] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 11 Jan '21

11 Jan '21

Prefix 'kernel ' and 'initrd ' of 'kernel_uri' and 'initrds_uri' are just used by 'boot.ipxe'. Need to make this independent to adapt other boot types. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 260b1ea..59a4c57 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -433,8 +433,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - JobHelper.service_path("#{linux_vmlinuz_path}") + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds @@ -497,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 1

[PATCH v4 compass-ci 1/4] scheduler: divide field 'kernel_params' into two fields
by Ren Wen 11 Jan '21

11 Jan '21

Original: 'kernel_params' is composed of common kernel params and kernel params for boot.ipxe. Now divide this field into 'kernel_params' and 'ipxe_kernel_params'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 11 +++-------- src/scheduler/kernel_params.cr | 9 ++++----- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..260b1ea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -100,6 +100,7 @@ class Job user_lkp_src kernel_uri kernel_params + ipxe_kernel_params docker_image kernel_version linux_vmlinuz_path @@ -488,14 +489,8 @@ class Job return temp_initrds end - private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + private def initrds_basename : Array(String) + return os_mount == "container" ? [] of String : get_initrds.map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..dea9215 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -33,6 +32,6 @@ class Job kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) + @hash["ipxe_kernel_params"] = JSON.parse(initrds_basename.to_json) end - end -- 2.23.0

1 1

[PATCH v4 compass-ci 4/4] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) - RESULT_WEBDAV_PORT (for file upload/download) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 0

[PATCH v3 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..6130246 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,21 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +199,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH compass-ci] fix: change tool's name to docker2rootfs
by Wang Chenglong 11 Jan '21

11 Jan '21

[Why] Change tool's name to docker2rootfs, like iso2rootfs. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/{docker-rootfs => docker2rootfs}/README.md | 0 container/{docker-rootfs => docker2rootfs}/common | 0 container/{docker-rootfs => docker2rootfs}/packages-to-install | 0 container/{docker-rootfs => docker2rootfs}/run | 0 container/{docker-rootfs => docker2rootfs}/setup-image | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (100%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 100% rename from container/docker-rootfs/common rename to container/docker2rootfs/common diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: reduce parameters
by Wang Yong 11 Jan '21

11 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 5 ++++- tests/cci-makepkg | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..8b1c73194 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..5af04b203 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci 2/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..3ad132e 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: only submit jobs of master branch for repos
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..02140d7 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,7 +31,10 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") + branch_master = branch.delete_prefix('refs/heads/') + next unless branch_master == 'master' + + real_argvs.push("upstream_branch=#{branch_master}") real_argvs.push("upstream_commit=#{commit_id}") tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig: optimize
by Luan Shengde 11 Jan '21

11 Jan '21

[why] when submitting jobs, the job will read date from /etc/compass-ci/defaults/*.yaml. the read date is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..cec6b2b 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH lkp-tests] tests/host-info: computing vm memory
by Wei Jihui 11 Jan '21

11 Jan '21

[why] it can not compute vm memory when vm memory less than 16G. [how] keep the round-up feature when memory bigger then 16G, and get actual memory when memory less then 16G. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/host-info | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index fe55d1ff..86b49cf9 100755 --- a/tests/host-info +++ b/tests/host-info @@ -5,7 +5,12 @@ get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') - memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + if [ `expr $memory_new \> 16` -eq 1 ]; then + memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + else + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + fi echo "memory: $memory" } -- 2.23.0

1 0

[PATCH compass-ci] lib/es-jobs-mapping.sh: add group_id
by Wu Zhende 11 Jan '21

11 Jan '21

Add "group_id" index, so we can search by group_id in es. "group_id" parameter identifies a group of jobs and is specified by users. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index b43625e..9c9f7b6 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -104,6 +104,9 @@ else "submit_id": { "type": "keyword" }, + "group_id": { + "type": "keyword" + }, "id": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: add submit_id
by Wu Zhende 11 Jan '21

11 Jan '21

A globally "submit_id" is automatically generated for each submit command and automatically added to jobs. Indicates the jobs of the same batch submitted by the command. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a92556fff..5273896b6 100755 --- a/sbin/submit +++ b/sbin/submit @@ -103,6 +103,9 @@ def find_jobfile(jobfile) exit 1 end +submit_id = %x(uuidgen).chomp +puts "submit_id=#{submit_id}" + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -113,6 +116,7 @@ ARGV.each do |jobfile| jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] + jobs['submit_id'] = submit_id jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

1 0

[PATCH compass-ci 3/3] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this because of the changes of 'initrds_uri' and 'kernel_uri' of the job. | field | before | now | |-------------+--------------+-------| | initrds_uri | initrd <uri> | <uri> | | kernel_uri | kernel <uri> | <uri> | Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..fe239d8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,8 +158,8 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " + response += "initrd " + Array(String).from_json(job.kernel_params).join("\ninitrd ") + "\n" + response += "kernel #{job.kernel_uri} " response += Array(String).from_json(job.kernel_params).join(" ") response += "\nboot\n" -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this to adapt the changes to job fields: - 'kernel_uri' - 'kernel_params' - 'initrds_uri' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..5e6044a 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,9 +158,13 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " - response += Array(String).from_json(job.kernel_params).join(" ") + + _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } + response += _initrds_uri.join("\n") + "\n" + + _kernel_params = ["kernel #{job.kernel_uri}"] + Array(String).from_json(job.kernel_params) + Array(String).from_json(job.ipxe_kernel_params) + response += _kernel_params.join(" ") + response += "\nboot\n" return response -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 11 Jan '21

11 Jan '21

Prefix 'kernel ' and 'initrd ' of 'kernel_uri' and 'initrds_uri' are just used by 'boot.ipxe'. Need to make this independent to adapt other boot types. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 260b1ea..59a4c57 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -433,8 +433,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - JobHelper.service_path("#{linux_vmlinuz_path}") + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds @@ -497,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: divide field 'kernel_params' into two fields
by Ren Wen 11 Jan '21

11 Jan '21

Original: 'kernel_params' is composed of common kernel params and kernel params for boot.ipxe. Now divide this field into 'kernel_params' and 'ipxe_kernel_params'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 11 +++-------- src/scheduler/kernel_params.cr | 9 ++++----- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..260b1ea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -100,6 +100,7 @@ class Job user_lkp_src kernel_uri kernel_params + ipxe_kernel_params docker_image kernel_version linux_vmlinuz_path @@ -488,14 +489,8 @@ class Job return temp_initrds end - private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + private def initrds_basename : Array(String) + return os_mount == "container" ? [] of String : get_initrds.map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..dea9215 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -33,6 +32,6 @@ class Job kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) + @hash["ipxe_kernel_params"] = JSON.parse(initrds_basename.to_json) end - end -- 2.23.0

1 0

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 11 Jan '21

11 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37d..3aa1f7f5 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -23,29 +23,18 @@ postgres_run_log() log_cmd sudo -u postgres "$@" } -init_postgresql() +start_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) +start_postgresql -init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

Re: [PATCH lkp-tests] tests: modify api-avx2neon
by Luan Shengde 11 Jan '21

11 Jan '21

> #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" >+ cd AvxToNeon/tests || exit is the directory is part of repo: $git_url? two line above => git clone "$git_url" || exit cd AvxToNeon/tests Thanks, Luan Shengde > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} > >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') >+ [ 0 -eq "$fail" ] || exit 1 >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

[PATCH compass-ci] providers/docker/run.sh: syncing docker time with localtime
by Wei Jihui 11 Jan '21

11 Jan '21

[why] docker time is GMT, docker time should be same with localtime. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- providers/docker/run.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 57a01c0..4d53db7 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -31,6 +31,7 @@ cmd=( -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro + -v /etc/localtime:/etc/localtime:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} -- 2.23.0

1 0

[PATCH v2 lkp-tests] pkg/api-avx2neon: fix x86 version build
by Wang Yong 11 Jan '21

11 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- pkg/api-avx2neon/PKGBUILD | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/pkg/api-avx2neon/PKGBUILD b/pkg/api-avx2neon/PKGBUILD index fac047add..30967beb7 100644 --- a/pkg/api-avx2neon/PKGBUILD +++ b/pkg/api-avx2neon/PKGBUILD @@ -1,3 +1,5 @@ +# add pkg_arch: x86_64 to build x86 version +# ref: Makefile pkgname=api-avx2neon pkgver=1 pkgrel=1 @@ -9,7 +11,12 @@ md5sums=('SKIP') build() { cd "$srcdir/AvxToNeon/tests" - make ARCH=$(arch) + + if [ "${pkg_arch}" == "aarch64" ]; then + make + elif [ "${pkg_arch}" == "x86_64" ]; then + make ARCH=x86 + fi } package() { -- 2.23.0

1 0

[PATCH v2 compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 11 Jan '21

11 Jan '21

before: output qemu-efi-aarch64 after: if arch=aarch64 -> output qemu-efi-aarch64 if arch=x86_64 -> output ovmf [why] for x86_64 virtual machine, qemu efi is ovmf, so use qemu efi according to different arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

1 0

[PATCH v7 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 11 Jan '21

11 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-aarch64/z9.291743 /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-x86_64/z9.291758 /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-riscv64/z9.291753 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..50098ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemus=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemus[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 11 Jan '21

11 Jan '21

only when images not exist or $action != "reboot", need to execute ./build and ./install, otherwise just execute ./first-run and ./start Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..6fd0ab2 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd $container [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

2 2

[PATCH v2 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 11 Jan '21

11 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] root@vm-8p64g /sys/class/net# ls */operstate docker0/operstate eth0/operstate lo/operstate vethf90a0ad/operstate root@vm-8p64g /sys/class/net# ls */carrier docker0/carrier eth0/carrier lo/carrier vethf90a0ad/carrier one qemu may have serial net interfaces in /sys/class/net, just one is 'up' or '1' means network is ok, it will not sleep and directly run function export_ip_mac(), but actually eth0 not ready, it can't get mac and ip, so use ping can ensure save enough time for eth0. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..df25e99e0 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER > /dev/null && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 11 Jan '21

11 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 11 Jan '21

11 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

Re: [PATCH lkp-tests] jobs: and ansible_test.yaml
by Liu Shaofei 11 Jan '21

11 Jan '21

^--- and => add ? In-Reply-To: <20210109124854.6869-1-ylzhangah(a)qq.com> On Sat, Jan 09, 2021 at 08:48:54PM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 9 +++++++++ > 1 file changed, 9 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..9adc4d71 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,9 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ username: #geerlingguy >+ rolename: #jenkins >+ ansible_host: #localhost >+ ansible_user: #root >+ ansible_python_interpreter: #/user/bin/python3 ^--- user => usr ? ansible_python_interpreter => ansible_python ? Thanks, Liushaofei >-- >2.23.0 >

2 1