- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container/add-software: add to clean hidden files in runtime dirs
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the first pkgbuild_repo if no '-git'
by Li Yuanchao 12 Jan '21

12 Jan '21

As new pkgbuild_repos don't have '-git' at the end, return the first one instead. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 8845b13..d765094 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -18,7 +18,7 @@ class AutoSubmit end def get_pkgbuild_repo(repo_array) - pkgbuild_repo = nil + pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| next unless repo =~ /-git$/ -- 2.23.0

2 1

[PATCH compass-ci] container: mount default config dir to delimiter container for submitting job
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/delimiter/start b/container/delimiter/start index 27e8d45..5d7037d 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -23,6 +23,7 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm 'delimiter' cmd = %w[ @@ -34,8 +35,9 @@ cmd = %w[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} - -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter -- 2.23.0

2 1

[PATCH v2 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 12 Jan '21

12 Jan '21

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..4ff4285 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,32 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +98,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

2 1

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 12 Jan '21

12 Jan '21

>+ def lftp_mirror >+ @result_root.delete_prefix!('/srv') >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' >+ url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" >+ system "lftp -c mirror #{url} >/dev/null 2>&1" >+ end >+ >+ def lftp(data) >+ if @result_root >+ lftp_mirror Seems it'll repeatedly call lftp on every new message? When do we start downloading? Ideally do it once after extract stats. >+ else >+ set_result_root(data) That else seems illogical. Typically we set a var before checking it. Maybe lftp_mirror will be called at the next message, however it's still illogical thing. Thanks, Fengguang >+ end >+ end >+ > def stop(data, web_socket, code = 1000, reason = 'normal') > @stop_query.each do |key, value| > return false unless data[key] == value >@@ -109,7 +136,7 @@ class Monitor > > @query.each do |k, v| > @query[k] = JSON.parse(v) >- rescue >+ rescue StandardError > end > query = @query.to_json > puts "query=>#{query}" >@@ -136,6 +163,7 @@ class Monitor > > output(data) > connect(data, ws) >+ lftp(data) > > stop(data, ws) if @action['stop'] > end >diff --git a/sbin/submit b/sbin/submit >index 05b3e86dd..a92556fff 100755 >--- a/sbin/submit >+++ b/sbin/submit >@@ -47,6 +47,10 @@ options = OptionParser.new do |opts| > actions << 'connect' > end > >+ opts.on('-l', '--lftp', 'lftp mirror job result') do >+ actions << 'lftp_result' >+ end >+ > opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do > opt_monitor = true > filter = YAML.load ARGV[0] if ARGV[0] >-- >2.23.0 >

1 0

[PATCH compass-ci] lib/job.cr: set sshr info when submit reproduce job
by Wu Zhende 12 Jan '21

12 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 6f344fe..218e2e1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -62,6 +62,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + set_sshr_info() return end end @@ -203,7 +204,7 @@ class Job end private def set_sshr_info - return unless @hash["ssh_pub_key"]? + return unless @hash.has_key?("ssh_pub_key") self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] -- 2.23.0

2 1

[PATCH lkp-tests 1/3] stats/crystal/memory--stat.cr: create memory--stat.cr with memory--stat
by Lu Kaiyi 12 Jan '21

12 Jan '21

create memory--stat.cr as same content with memory--stat Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/memory--stat.cr diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr new file mode 100644 index 000000000..1a39e4075 --- /dev/null +++ b/stats/crystal/memory--stat.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby + +# time: 123456 +# cgroup: pmbench.1 +# cache 0 + +def parse + cg = 'nocg' + $stdin.each_line do |line| + case line + when /^time:/ + puts line + when /^cgroup: (\S+)/ + cg = $1 + when /(\S+)\s*(\S+)/ + k, v, = line.split + puts "#{cg}.#{k}: #{v}" + end + end +end + +parse -- 2.23.0

2 1

[PATCH v3 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..e3dfd17 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo SCHED_PORT: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 0

[PATCH v2 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..6acfc2e 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo sched_port: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 1

[PATCH compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..03bb061 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,16 @@ run_docker() done } +kill_process() +{ + multi_qemu=`ps -ef |grep multi-qemu |grep "$USER" |awk '{print $2}'` + multi_docker=`ps -ef |grep multi-docker |grep "$USER" |awk '{print $2}'` + kill -9 $multi_qemu + kill -9 $multi_docker +} + run_qemu run_docker wait + +kill_process -- 2.23.0

3 2

[PATCH compass-ci] sbin/result2stats: support merge yaml file
by Lu Weitao 12 Jan '21

12 Jan '21

[Why] user may extract $program result to $program.yaml Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index e603763..8444334 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,19 +39,21 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') + when /\.yaml$/ + File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if monitor == 'stats' # stats.json already created? + next if %w[stats job].include?(monitor) extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] stats/crystal/memory--stat.cr: fix some errors manually
by Lu Kaiyi 12 Jan '21

12 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index da7ac0d86..f45ab5fb1 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -6,14 +6,14 @@ def parse cg = "nocg" - $stdin.each_line do |line| + STDIN.each_line do |line| case line when /^time:/ puts line when /^cgroup: (\S+)/ cg = $1 when /(\S+)\s*(\S+)/ - k, v, = line.split + k, v = line.split puts "#{cg}.#{k}: #{v}" end end -- 2.23.0

1 0

[PATCH lkp-tests 2/3] stats/crystal/memory--stat.cr: do first conversion by ruby2crystal tools
by Lu Kaiyi 12 Jan '21

12 Jan '21

use the ruby2crystal tools do first conversion like below: /c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod memory--stat.cr /c/lkp-tests/tools/ruby2crystal.sh memory--stat.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index 1a39e4075..da7ac0d86 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -1,11 +1,11 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # time: 123456 # cgroup: pmbench.1 # cache 0 def parse - cg = 'nocg' + cg = "nocg" $stdin.each_line do |line| case line when /^time:/ -- 2.23.0

1 0

[PATCH compass-ci] addsoftware: add to clean hidden files in runtime dir
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

2 4

[PATCH v1 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..ea0c259 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,33 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + if @hash.has_key?(item) + pkg_style = item + break + end end + return nil unless pkg_style + + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +287,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

$lab will be used in $CCI_SRC/sbin/build-my-info.rb, so add it. can see commit 66b58a7be3cf371c74946950ebe9dc73efd9a16e Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..69b31cd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -43,6 +43,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +lab: nolab EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH v6 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 12 Jan '21

12 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v6 compass-ci 6/7] lib/mail-post: add email counts/mapping checks
by Luan Shengde 12 Jan '21

12 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v6 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 12 Jan '21

12 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v6 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 12 Jan '21

12 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v6 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 12 Jan '21

12 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v6 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v6 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 12 Jan '21

12 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v2 compass-ci] container/auto-submit: fix submit job failed
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user intall compass-ci at their local server, need to verify when submit job in auto-submit container, so store autosubmit_info to ES, and mount yaml file in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- lib/build_my_info_client.rb | 4 ++-- sbin/build-my-info.rb | 4 ++++ 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..d0d4b77 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,12 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = %x(uuidgen).chomp) @lab = lab || 'nolab' @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..d68932e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -31,3 +31,7 @@ options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info + +autosubmit_info = YAML.load_file('/etc/compass-ci/account/autosubmit.yaml') +build_autosubmit_info = BuildMyInfo.new(autosubmit_info['my_email'], autosubmit_info['my_name'], autosubmit_info['lab'], autosubmit_info['my_token']) +build_autosubmit_info.store_account_info -- 2.23.0

2 2

[PATCH v3 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= export_mac_ip: PUB_NIC=$(ip route get 1.2.3.4 | awk '{print $5; exit}') PUB_IP=$( ip route get 1.2.3.4 | awk '{print $7; exit}') network_ok: [ "$(cat $i/operstate)" = 'up' ]&& return 0 when cat /sys/class/net/eth0/operstate is up, network is ok, then it will not sleep and immediately execute export_ip_mac(), but actually route is not ready, it can't get mac and ip by search route, so use ping can ensure save enough time for route. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..df25e99e0 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER > /dev/null && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

3 4

[PATCH compass-ci] sparrow/3-code: add RESULT_WEBDAV_PORT field
by Xiao Shenwei 12 Jan '21

12 Jan '21

[why] $LKP_SRC/lib/upload.sh will upload some result file by result-webdav container [how] display definition in default profile Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..f2366c2 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -19,6 +19,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} : ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_PORT:=3080} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -43,6 +44,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..1c6eb1a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,10 +249,37 @@ class Job set_upload_dirs() end + def get_pkg_common_dir + common_dir = "" + + if @hash["cci-makepkg"]? + pkg_style = "cci-makepkg" + elsif @hash["cci-depends"]? + pkg_style = "cci-depends" + elsif @hash["build-pkg"]? + pkg_style = "build-pkg" + else + pkg_style = nil + end + + if pkg_style + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end + + return common_dir + end + def get_package_dir if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + common_dir = get_pkg_common_dir end if @hash["cci-makepkg"]? -- 2.23.0

2 2

[PATCH lkp-tests] reset-tbox to reboot machine testbox
by Zhang Dewan 11 Jan '21

11 Jan '21

[why] sumbit a job to reboot machine testbox is better. [how] cp ~/lkp-test/jobs/reset-tbox-yaml . submit reset-tbox.yaml host_name="full-name-of-machine-testbox" Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 17 +++++++++++++++++ tests/reset-tbox.yaml | 12 ++++++++++++ 3 files changed, 39 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox create mode 100644 tests/reset-tbox.yaml diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..9c3f4f01 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +testcase: reset-tbox + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset + +host_name: +reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..b0f645d7 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,17 @@ +#!/bin/bash +# host_name + +source "/etc/lab/.account.info" +hosts_file=/etc/lab/hosts_file + +# require $host_name +host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + +if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then + echo ">>> exec ipmitool commamd!" + ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + [ $? -eq 0 ] && echo "succeed!" +else + echo "specify host cannot found0!" + exit +fi diff --git a/tests/reset-tbox.yaml b/tests/reset-tbox.yaml new file mode 100644 index 00000000..f557085f --- /dev/null +++ b/tests/reset-tbox.yaml @@ -0,0 +1,12 @@ +short_description: > + To reboot machine testbox like taishan200-2280-2s64p-256g--aN etc. + +description: | + reset-tbox executed by a special docker which running on ibmc-mgnt, where you can access all server's iBMC-os by ipmitool, and then reboot the host. + +homepage: + +parameters: + host_name: see it on the first line about 'latbox aN' + +results: -- 2.23.0

3 4

[PATCH v4 compass-ci 3/4] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this to adapt the changes to job fields: - 'kernel_uri' - 'kernel_params' - 'initrds_uri' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..5e6044a 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,9 +158,13 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " - response += Array(String).from_json(job.kernel_params).join(" ") + + _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } + response += _initrds_uri.join("\n") + "\n" + + _kernel_params = ["kernel #{job.kernel_uri}"] + Array(String).from_json(job.kernel_params) + Array(String).from_json(job.ipxe_kernel_params) + response += _kernel_params.join(" ") + response += "\nboot\n" return response -- 2.23.0

1 1

[PATCH v4 compass-ci 2/4] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 11 Jan '21

11 Jan '21

Prefix 'kernel ' and 'initrd ' of 'kernel_uri' and 'initrds_uri' are just used by 'boot.ipxe'. Need to make this independent to adapt other boot types. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 260b1ea..59a4c57 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -433,8 +433,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - JobHelper.service_path("#{linux_vmlinuz_path}") + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds @@ -497,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 1

[PATCH v4 compass-ci 1/4] scheduler: divide field 'kernel_params' into two fields
by Ren Wen 11 Jan '21

11 Jan '21

Original: 'kernel_params' is composed of common kernel params and kernel params for boot.ipxe. Now divide this field into 'kernel_params' and 'ipxe_kernel_params'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 11 +++-------- src/scheduler/kernel_params.cr | 9 ++++----- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..260b1ea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -100,6 +100,7 @@ class Job user_lkp_src kernel_uri kernel_params + ipxe_kernel_params docker_image kernel_version linux_vmlinuz_path @@ -488,14 +489,8 @@ class Job return temp_initrds end - private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + private def initrds_basename : Array(String) + return os_mount == "container" ? [] of String : get_initrds.map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..dea9215 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -33,6 +32,6 @@ class Job kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) + @hash["ipxe_kernel_params"] = JSON.parse(initrds_basename.to_json) end - end -- 2.23.0

1 1

[PATCH v4 compass-ci 4/4] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) - RESULT_WEBDAV_PORT (for file upload/download) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 0

[PATCH v3 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..6130246 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,21 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +199,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH compass-ci] fix: change tool's name to docker2rootfs
by Wang Chenglong 11 Jan '21

11 Jan '21

[Why] Change tool's name to docker2rootfs, like iso2rootfs. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/{docker-rootfs => docker2rootfs}/README.md | 0 container/{docker-rootfs => docker2rootfs}/common | 0 container/{docker-rootfs => docker2rootfs}/packages-to-install | 0 container/{docker-rootfs => docker2rootfs}/run | 0 container/{docker-rootfs => docker2rootfs}/setup-image | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (100%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 100% rename from container/docker-rootfs/common rename to container/docker2rootfs/common diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: reduce parameters
by Wang Yong 11 Jan '21

11 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 5 ++++- tests/cci-makepkg | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..8b1c73194 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..5af04b203 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci 2/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..3ad132e 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: only submit jobs of master branch for repos
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..02140d7 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,7 +31,10 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") + branch_master = branch.delete_prefix('refs/heads/') + next unless branch_master == 'master' + + real_argvs.push("upstream_branch=#{branch_master}") real_argvs.push("upstream_commit=#{commit_id}") tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig: optimize
by Luan Shengde 11 Jan '21

11 Jan '21

[why] when submitting jobs, the job will read date from /etc/compass-ci/defaults/*.yaml. the read date is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..cec6b2b 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH lkp-tests] tests/host-info: computing vm memory
by Wei Jihui 11 Jan '21

11 Jan '21

[why] it can not compute vm memory when vm memory less than 16G. [how] keep the round-up feature when memory bigger then 16G, and get actual memory when memory less then 16G. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/host-info | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index fe55d1ff..86b49cf9 100755 --- a/tests/host-info +++ b/tests/host-info @@ -5,7 +5,12 @@ get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') - memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + if [ `expr $memory_new \> 16` -eq 1 ]; then + memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + else + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + fi echo "memory: $memory" } -- 2.23.0

1 0

[PATCH compass-ci] lib/es-jobs-mapping.sh: add group_id
by Wu Zhende 11 Jan '21

11 Jan '21

Add "group_id" index, so we can search by group_id in es. "group_id" parameter identifies a group of jobs and is specified by users. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index b43625e..9c9f7b6 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -104,6 +104,9 @@ else "submit_id": { "type": "keyword" }, + "group_id": { + "type": "keyword" + }, "id": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: add submit_id
by Wu Zhende 11 Jan '21

11 Jan '21

A globally "submit_id" is automatically generated for each submit command and automatically added to jobs. Indicates the jobs of the same batch submitted by the command. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a92556fff..5273896b6 100755 --- a/sbin/submit +++ b/sbin/submit @@ -103,6 +103,9 @@ def find_jobfile(jobfile) exit 1 end +submit_id = %x(uuidgen).chomp +puts "submit_id=#{submit_id}" + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -113,6 +116,7 @@ ARGV.each do |jobfile| jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] + jobs['submit_id'] = submit_id jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

1 0

[PATCH compass-ci 3/3] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this because of the changes of 'initrds_uri' and 'kernel_uri' of the job. | field | before | now | |-------------+--------------+-------| | initrds_uri | initrd <uri> | <uri> | | kernel_uri | kernel <uri> | <uri> | Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..fe239d8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,8 +158,8 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " + response += "initrd " + Array(String).from_json(job.kernel_params).join("\ninitrd ") + "\n" + response += "kernel #{job.kernel_uri} " response += Array(String).from_json(job.kernel_params).join(" ") response += "\nboot\n" -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this to adapt the changes to job fields: - 'kernel_uri' - 'kernel_params' - 'initrds_uri' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..5e6044a 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,9 +158,13 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " - response += Array(String).from_json(job.kernel_params).join(" ") + + _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } + response += _initrds_uri.join("\n") + "\n" + + _kernel_params = ["kernel #{job.kernel_uri}"] + Array(String).from_json(job.kernel_params) + Array(String).from_json(job.ipxe_kernel_params) + response += _kernel_params.join(" ") + response += "\nboot\n" return response -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 11 Jan '21

11 Jan '21

Prefix 'kernel ' and 'initrd ' of 'kernel_uri' and 'initrds_uri' are just used by 'boot.ipxe'. Need to make this independent to adapt other boot types. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 260b1ea..59a4c57 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -433,8 +433,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - JobHelper.service_path("#{linux_vmlinuz_path}") + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds @@ -497,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: divide field 'kernel_params' into two fields
by Ren Wen 11 Jan '21

11 Jan '21

Original: 'kernel_params' is composed of common kernel params and kernel params for boot.ipxe. Now divide this field into 'kernel_params' and 'ipxe_kernel_params'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 11 +++-------- src/scheduler/kernel_params.cr | 9 ++++----- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..260b1ea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -100,6 +100,7 @@ class Job user_lkp_src kernel_uri kernel_params + ipxe_kernel_params docker_image kernel_version linux_vmlinuz_path @@ -488,14 +489,8 @@ class Job return temp_initrds end - private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + private def initrds_basename : Array(String) + return os_mount == "container" ? [] of String : get_initrds.map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..dea9215 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -33,6 +32,6 @@ class Job kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) + @hash["ipxe_kernel_params"] = JSON.parse(initrds_basename.to_json) end - end -- 2.23.0

1 0

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 11 Jan '21

11 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37d..3aa1f7f5 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -23,29 +23,18 @@ postgres_run_log() log_cmd sudo -u postgres "$@" } -init_postgresql() +start_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) +start_postgresql -init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

Re: [PATCH lkp-tests] tests: modify api-avx2neon
by Luan Shengde 11 Jan '21

11 Jan '21

> #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" >+ cd AvxToNeon/tests || exit is the directory is part of repo: $git_url? two line above => git clone "$git_url" || exit cd AvxToNeon/tests Thanks, Luan Shengde > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} > >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') >+ [ 0 -eq "$fail" ] || exit 1 >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

[PATCH compass-ci] providers/docker/run.sh: syncing docker time with localtime
by Wei Jihui 11 Jan '21

11 Jan '21

[why] docker time is GMT, docker time should be same with localtime. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- providers/docker/run.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 57a01c0..4d53db7 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -31,6 +31,7 @@ cmd=( -v $CCI_SRC:/c/compass-ci:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro + -v /etc/localtime:/etc/localtime:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} -- 2.23.0

1 0

[PATCH v2 lkp-tests] pkg/api-avx2neon: fix x86 version build
by Wang Yong 11 Jan '21

11 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- pkg/api-avx2neon/PKGBUILD | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/pkg/api-avx2neon/PKGBUILD b/pkg/api-avx2neon/PKGBUILD index fac047add..30967beb7 100644 --- a/pkg/api-avx2neon/PKGBUILD +++ b/pkg/api-avx2neon/PKGBUILD @@ -1,3 +1,5 @@ +# add pkg_arch: x86_64 to build x86 version +# ref: Makefile pkgname=api-avx2neon pkgver=1 pkgrel=1 @@ -9,7 +11,12 @@ md5sums=('SKIP') build() { cd "$srcdir/AvxToNeon/tests" - make ARCH=$(arch) + + if [ "${pkg_arch}" == "aarch64" ]; then + make + elif [ "${pkg_arch}" == "x86_64" ]; then + make ARCH=x86 + fi } package() { -- 2.23.0

1 0

[PATCH v2 compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 11 Jan '21

11 Jan '21

before: output qemu-efi-aarch64 after: if arch=aarch64 -> output qemu-efi-aarch64 if arch=x86_64 -> output ovmf [why] for x86_64 virtual machine, qemu efi is ovmf, so use qemu efi according to different arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

1 0

[PATCH v7 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 11 Jan '21

11 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-aarch64/z9.291743 /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-x86_64/z9.291758 /srv/result/host-info/2021-01-11/vm-2p8g/debian-sid-riscv64/z9.291753 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..50098ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemus=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemus[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 11 Jan '21

11 Jan '21

only when images not exist or $action != "reboot", need to execute ./build and ./install, otherwise just execute ./first-run and ./start Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..6fd0ab2 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd $container [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

2 2

[PATCH v2 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 11 Jan '21

11 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] root@vm-8p64g /sys/class/net# ls */operstate docker0/operstate eth0/operstate lo/operstate vethf90a0ad/operstate root@vm-8p64g /sys/class/net# ls */carrier docker0/carrier eth0/carrier lo/carrier vethf90a0ad/carrier one qemu may have serial net interfaces in /sys/class/net, just one is 'up' or '1' means network is ok, it will not sleep and directly run function export_ip_mac(), but actually eth0 not ready, it can't get mac and ip, so use ping can ensure save enough time for eth0. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..df25e99e0 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER > /dev/null && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 11 Jan '21

11 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 11 Jan '21

11 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

Re: [PATCH lkp-tests] jobs: and ansible_test.yaml
by Liu Shaofei 11 Jan '21

11 Jan '21

^--- and => add ? In-Reply-To: <20210109124854.6869-1-ylzhangah(a)qq.com> On Sat, Jan 09, 2021 at 08:48:54PM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 9 +++++++++ > 1 file changed, 9 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..9adc4d71 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,9 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ username: #geerlingguy >+ rolename: #jenkins >+ ansible_host: #localhost >+ ansible_user: #root >+ ansible_python_interpreter: #/user/bin/python3 ^--- user => usr ? ansible_python_interpreter => ansible_python ? Thanks, Liushaofei >-- >2.23.0 >

2 1

[PATCH compass-ci] fix: change tool's name to docker2rootfs
by Wang Chenglong 11 Jan '21

11 Jan '21

[Why] Change tool's name to docker2rootfs, like iso2rootfs. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/{docker-rootfs => docker2rootfs}/README.md | 0 container/{docker-rootfs => docker2rootfs}/common | 0 container/{docker-rootfs => docker2rootfs}/packages-to-install | 0 container/{docker-rootfs => docker2rootfs}/run | 0 container/{docker-rootfs => docker2rootfs}/setup-image | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (100%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 100% rename from container/docker-rootfs/common rename to container/docker2rootfs/common diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: fix submit job failed
by Liu Yinsi 11 Jan '21

11 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user intall compass-ci at their local server, need to verify when submit job in auto-submit container, so store autosubmit_info to ES, and mount yaml file in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- sbin/build-my-info.rb | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..ea581ca 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -31,3 +31,7 @@ options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info + +autosubmit_info = YAML.load_file('/etc/compass-ci/account/autosubmit.yaml') +build_autosubmit_info = BuildMyInfo.new(autosubmit_info['my_email'], autosubmit_info['my_name'], autosubmit_info['lab']) +build_autosubmit_info.store_account_info -- 2.23.0

1 0

[PATCH v7 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 11 Jan '21

11 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v7 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 11 Jan '21

11 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

Re: [PATCH lkp-tests] jobs: and ansible_test.yaml
by Bai Jing 11 Jan '21

11 Jan '21

^^ add Thanks, Baijing Reply-To: In-Reply-To: <20210109124854.6869-1-ylzhangah(a)qq.com> On Sat, Jan 09, 2021 at 08:48:54PM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 9 +++++++++ > 1 file changed, 9 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..9adc4d71 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,9 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ username: #geerlingguy >+ rolename: #jenkins >+ ansible_host: #localhost >+ ansible_user: #root >+ ansible_python_interpreter: #/user/bin/python3 >-- >2.23.0 >

2 1

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 11 Jan '21

11 Jan '21

This script can be invoked to run the ansible command to deploy ansible-roles which you specified in your environment. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..09e5ca9e --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,59 @@ +#!/bin/bash +# username +# rolename +# ansible_host +# ansible_user +# ansible_python_interpreter + +[ -n "$username" ] || echo "username is empty" +[ -n "$rolename" ] || echo "rolename is empty" +[ -n "$ansible_host" ] || $ansible_host=localhost +[ -n "$ansible_user" ] || $ansible_user=root +[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 + +build_mutual_trust() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa + cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys + ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' +} + +install_ansible() +{ + pip3 install ansible + ansible localhost -m ping + #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko + [ $? -eq 0 ] && echo "Ansible is successfully deployed!" +} + +download_ansible_role() +{ + ansible-galaxy install $username.$rolename + [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" +} + +ansible_playbook_run() +{ + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible + cat <<-EOF > "$ansible_path"/hosts + $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter + EOF + [ $? -eq 0 ] && echo "Hostfile has been generated!" + + cat <<-EOF > "$ansible_path"/$rolename.yml + - hosts: $ansible_host + remote_user: $ansible_user + roles: + - role: $username.$rolename + EOF + [ $? -eq 0 ] && echo "Playbook has been generated!" + + ansible-playbook "$ansible_path"/$rolename.yml + [ $? -eq 0 ] && echo "Your playbook is running successfully!" +} + +#build_mutual_trust +install_ansible +download_ansible_role +ansible_playbook_run -- 2.23.0

5 10

[PATCH lkp-tests] sbin/submit: modify job_id to job id
by Hu Xuejiao 11 Jan '21

11 Jan '21

[why] Because we use "es-find (id='')" to search id-information, but when submit, it is (job_id=''). so it is recommended that the value be the same as the ID Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a92556ff..448d6243 100755 --- a/sbin/submit +++ b/sbin/submit @@ -164,10 +164,10 @@ ARGV.each do |jobfile| JSON.parse(messages).each do |msg| if msg['message'].empty? job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job_id=#{msg['job_id']}") + puts("submit #{jobfile}, got job id=#{msg['job_id']}") else opt_monitor = false - puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") end end end -- 2.23.0

2 1

[PATCH lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 10 Jan '21

10 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..9adc4d71 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,9 @@ +suite: ansible_test +category: functional + +ansible_test: + username: #geerlingguy + rolename: #jenkins + ansible_host: #localhost + ansible_user: #root + ansible_python_interpreter: #/user/bin/python3 -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends: add depends for openeuler_docker test
by Bai Jing 09 Jan '21

09 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- distro/depends/openeuler_docker | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/openeuler_docker diff --git a/distro/depends/openeuler_docker b/distro/depends/openeuler_docker new file mode 100644 index 000000000..b31074ad9 --- /dev/null +++ b/distro/depends/openeuler_docker @@ -0,0 +1,3 @@ +docker.io +wget +git -- 2.23.0

1 0

[PATCH compass-ci] qemu/kvm.sh: modify the prompt information of kernel
by Hu Xuejiao 09 Jan '21

09 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..577bca1 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -59,7 +59,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" + echo "The kernel does not exist" exit } -- 2.23.0

1 0

[PATCH compass-ci 3/3] build-my-info: fix name mismatches
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index d0ee46e..fd75f51 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -15,11 +15,11 @@ option = { options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| - option[:email] = email + option[:my_email] = email end opts.on('-n name', 'my_name') do |name| - option[:name] = name + option[:my_name] = name end opts.on('-l lab', 'lab') do |lab| -- 2.23.0

1 0

[PATCH compass-ci 2/3] build-my-info: set default values
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 0b16884..d0ee46e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -7,7 +7,12 @@ require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' require 'optparse' -option = {} +option = { + my_name: `git config --global user.name`.chomp, + my_email: `git config --global user.email`.chomp, + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp +} + options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email -- 2.23.0

1 0

[PATCH compass-ci 1/3] build-my-info: fix white space style
by Wu Fengguang 09 Jan '21

09 Jan '21

Layout/SpaceAroundOperators: Surrounding space missing for operator `=`. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index cd824c6..0b16884 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -8,7 +8,7 @@ require_relative '../lib/build_my_info_client' require 'optparse' option = {} -options= OptionParser.new do |opts| +options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email end -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

http/https cache proxy and request forwarding processing Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 118 +++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..2e83286 --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,118 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_block { + local http = require "resty.http" + local httpc = http.new() + uri = ngx.var.request_uri + host = ngx.var.http_host + if string.find(host, "11333") then + h="" + else + h="/"..host + end + path_file = "/srv/initrd/tar/"..h..uri + + local f = io.open(path_file) + if f then + url_forward = "http://127.0.0.1:80/tar/"..h..uri + else + url_forward = "http:/"..h..uri + end + + res, error = httpc:request_uri(url_forward) + if res then + ngx.status = res.status + for k, v in pairs(res.headers) do + if k ~= "Transfer-Encoding" and k ~= "Connection" then + ngx.header[k] = v + end + end + ngx.say(res.body) + else + ngx.exit(ngx.HTTP_FORBIDDEN) + end + httpc:close() + + } + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add start script
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/start | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 container/openresty-proxy-cache/start diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start new file mode 100755 index 0000000..a1f095d --- /dev/null +++ b/container/openresty-proxy-cache/start @@ -0,0 +1,22 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm openresty-proxy-cache + +cmd=( + docker run -d + --name openresty-proxy-cache + --restart=always + -p 11333:80 + -p 11334:443 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /srv/initrd/tar:/srv/initrd/tar:ro + -v /etc/localtime:/etc/localtime:ro + openresty:proxy_cache +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lua_script/forward.lua: http/https request forwarding processing
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- .../lua_script/forward.lua | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/openresty-proxy-cache/lua_script/forward.lua diff --git a/container/openresty-proxy-cache/lua_script/forward.lua b/container/openresty-proxy-cache/lua_script/forward.lua new file mode 100755 index 0000000..bd54742 --- /dev/null +++ b/container/openresty-proxy-cache/lua_script/forward.lua @@ -0,0 +1,32 @@ +local http = require "resty.http" +local httpc = http.new() +uri = ngx.var.request_uri +host = ngx.var.http_host +if string.find(host, "11333") then + h="" +else + h="/"..host +end +path_file = "/srv/initrd/tar/"..h..uri + +local f = io.open(path_file) +if f then + url_forward = "http://127.0.0.1:80/tar/"..h..uri +else + url_forward = "http:/"..h..uri +end + +res, error = httpc:request_uri(url_forward) +if res then + ngx.status = res.status + for k, v in pairs(res.headers) do + if k ~= "Transfer-Encoding" and k ~= "Connection" then + ngx.header[k] = v + end + end + ngx.say(res.body) +else + ngx.exit(ngx.HTTP_FORBIDDEN) +end + +httpc:close() -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..e3f11e0 --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_file lua_script/forward.lua; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH lkp-tests] hosts/dc-*g: add nr_cpu
by Hu Xuejiao 08 Jan '21

08 Jan '21

[why] Because when job.rb(expand_params) deal with params/nr_threads [ ($1.to_f * nr_cpu / 100).to_i ], it will handle the exception --"rescue KeyError"-- in running docker [so] i need add the nr_cpu in dc-*g Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- hosts/dc-1g | 1 + hosts/dc-2g | 1 + hosts/dc-4g | 1 + hosts/dc-8g | 1 + 4 files changed, 4 insertions(+) diff --git a/hosts/dc-1g b/hosts/dc-1g index ec1e59b9..24ff4227 100644 --- a/hosts/dc-1g +++ b/hosts/dc-1g @@ -1 +1,2 @@ +nr_cpu: 1 memory: 1g diff --git a/hosts/dc-2g b/hosts/dc-2g index b9509a2c..82c2522e 100644 --- a/hosts/dc-2g +++ b/hosts/dc-2g @@ -1 +1,2 @@ +nr_cpu: 2 memory: 2g diff --git a/hosts/dc-4g b/hosts/dc-4g index 52e0636d..30353733 100644 --- a/hosts/dc-4g +++ b/hosts/dc-4g @@ -1 +1,2 @@ +nr_cpu: 4 memory: 4g diff --git a/hosts/dc-8g b/hosts/dc-8g index 333d6250..470f3094 100644 --- a/hosts/dc-8g +++ b/hosts/dc-8g @@ -1 +1,2 @@ +nr_cpu: 8 memory: 8g -- 2.23.0

1 0

[PATCH compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..79aa10e --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_file lua_script/forward.lua; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH compass-ci] container/openresty-proxy-cache: add start script
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/openresty-proxy-cache/start diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start new file mode 100755 index 0000000..b946426 --- /dev/null +++ b/container/openresty-proxy-cache/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm openresty-proxy-cache + +cmd=( + docker run -d + --name openresty-proxy-cache + --restart=always + -p 11333:80 + -p 11334:443 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v $PWD/lua_script/forward.lua:/usr/local/openresty/nginx/lua_script/forward.lua + -v /srv/initrd/tar:/srv/initrd/tar:ro + -v /etc/localtime:/etc/localtime:ro + openresty:proxy_cache +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 08 Jan '21

08 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v5 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 08 Jan '21

08 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v5 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 08 Jan '21

08 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 08 Jan '21

08 Jan '21

example: name => email_addr logo => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..4c068b0 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# email_address_r1 => email_address_d +# email_address_r2 => email_address_d +# ... +# can set multi keys for an email address, the key can be a name, +# an email, or some logo else +# case add the key is added to email address bar, the key will be +# transferred to the destination email address +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 08 Jan '21

08 Jan '21

daily timing work: reset email rate limit queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..62f27ea --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 08 Jan '21

08 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v5 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v4 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

2 2

review star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our review star, thanks Jihui! jimmy@crystal ~/todo/bin% ./review-star 7 Review email records in the last 7 days total review paragraph point people 53 31 68 26 Wu Fengguang 87 15 18 10 Wei Jihui 37 10 20 9 Lu Kaiyi 32 4 11 8 Yu Chuan 58 15 33 7 Luan Shengde 24 5 8 6 Xiao Shenwei 20 11 12 6 Xu Xijian 10 5 11 5 Wu Zhende 32 3 3 4 Hu Xuejiao 19 8 5 4 Liu Shaofei 22 9 9 4 Ren Wen 28 2 2 3 Cao Xueliang 29 6 5 2 Cui Yili 10 8 8 2 Wang Yong 17 2 3 2 Li Ping 3 3 5 2 Zhang Dewan 12 4 7 1 Lin Jiaxin 23 4 6 1 Wang Chenglong 10 2 2 1 Li Yuanchao 13 1 1 1 Lu Weitao 5 1 1 1 Zhang Yale 33 2 2 1 Liu Yinsi 6 1 2 0 Bai Jing

1 0

git count
by Wu Fengguang 08 Jan '21

08 Jan '21

Yu Chuan is our code star again, thanks Yu Chuan! 10 Yu Chuan 6 Wei Jihui 6 Wu Fengguang 6 Li Yuanchao 5 Liu Shaofei 4 Xu Xijian 4 Hu Xuejiao 4 Lu Weitao 4 Wu Zhende 4 Xiao Shenwei 4 Lu Kaiyi 3 Wang Chenglong 3 Li Ping 2 Bai Jing 2 Cao Xueliang 2 Cui Yili 2 Lin Jiaxin 1 Zhang Yale 1 Luan Shengde 0 Tong Qunfeng 0 Du Kaitian 0 Liu Yinsi 0 Sun Yukui 0 Zhang Yu 0 Wang Yong 0 Ren Wen 0 Zhang Dewan 0 Shi Zhichao

1 0

help star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our help star, thanks Jihui! 15.2 Wu Fengguang 13.0 Wei Jihui 12.1 Yu Chuan 11.6 Xiao Shenwei 9.6 Lu Weitao 8.3 Ren Wen 7.5 Cao Xueliang 7.3 Wu Zhende 6.8 Zhang Yale 6.6 Wang Yong 6.1 Zhang Dewan 6.0 Liu Shaofei 4.8 Liu Yinsi 4.3 Xu Xijian 3.6 Wang Chenglong 3.3 Du Kaitian 2.8 Li Ping 2.7 Lin Jiaxin 2.3 Bai Jing 2.2 Cui Yili 2.0 Luan Shengde 1.3 Hu Xuejiao 0.7 Lu Kaiyi 0.0 Tong Qunfeng 0.0 Li Yuanchao 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH lkp-tests] adaptation/openeuler: add dependency of pgbench
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/openeuler | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 09120489..1938a8aa 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -295,3 +295,4 @@ ruby-full: ruby ruby-devel rubygem-rdoc linux-base: openssh-server: openssh openssh-client: openssh +postgresql-server: postgresql -- 2.23.0

2 1

[PATCH lkp-tests] tests/ebizzy: modify the value of iterations
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/ebizzy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ebizzy b/tests/ebizzy index dcc301871..394bc09ba 100755 --- a/tests/ebizzy +++ b/tests/ebizzy @@ -19,7 +19,7 @@ duration=${duration%%s} [ -n "$mmap" ] && opt_mmap=" -m " [ -n "$linear" ] && opt_linear=" -z " - +[[ "$iterations" =~ ^[0-9]{3}x$ ]] && iterations=100 for i in $(seq 1 $iterations) do echo Iteration: $i -- 2.23.0

3 2

[PATCH v2 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 08 Jan '21

08 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 08 Jan '21

08 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..cfc70ec --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|logo', '--name name|email|logo', 'appoint a name|email|logo to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || nil + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH compass-ci] uniform constant naming
by Lu Weitao 08 Jan '21

08 Jan '21

[Why] most file use "SEND_MAIL_HOST/PORT" [How] sed -i "s/[^_]MAIL_HOST/SEND_MAIL_HOST/g" `grep MAIL_HOST -rl compass-ci/*` sed -i "s/[^_]MAIL_PORT/SEND_MAIL_PORT/g" `grep MAIL_PORT -rl compass-ci/*` Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/delimiter/start | 4 ++-- container/extract-stats/start | 4 ++-- lib/constants.rb | 4 ++-- lib/mail_client.rb | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index 5d7037d..cb44d84 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -9,8 +9,8 @@ require_relative '../defconfig.rb' names = Set.new %w[ ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT MONITOR_HOST MONITOR_PORT TASKQUEUE_HOST diff --git a/container/extract-stats/start b/container/extract-stats/start index 74cf917..6510564 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,8 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT LAB ] diff --git a/lib/constants.rb b/lib/constants.rb index c3c7543..a274adc 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +SEND_MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 diff --git a/lib/mail_client.rb b/lib/mail_client.rb index f63e65d..6f5a099 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -8,8 +8,8 @@ require_relative 'constants.rb' # mail client class class MailClient - HOST = (ENV.key?('MAIL_HOST') ? ENV['MAIL_HOST'] : MAIL_HOST) - PORT = (ENV.key?('MAIL_PORT') ? ENV['MAIL_PORT'] : MAIL_PORT).to_i + HOST = (ENV.key?('SEND_MAIL_HOST') ? ENV['SEND_MAIL_HOST'] : SEND_MAIL_HOST) + PORT = (ENV.key?('SEND_MAIL_PORT') ? ENV['SEND_MAIL_PORT'] : SEND_MAIL_PORT).to_i def initialize(host = HOST, port = PORT) @host = host @port = port -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/auto_submit.yaml: add/update linux repos
by Lin Jiaxin 08 Jan '21

08 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit.yaml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 69b73ff..22172eb 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,7 +4,19 @@ c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml l/linux/linux: -- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/linux-next: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/openeuler-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/debian-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/archlinux-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

3 2

[PATCH lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 08 Jan '21

08 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [ 75.489958] Kernel tests: Boot OK! ... [ 82.149060] sysrq: Resetting [why] check network ok or not by checking network adapter is not accurate, use ping ip more accurate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3bd..b7795b40a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER | grep -q ttl && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 08 Jan '21

08 Jan '21

Prefix 'kernel ' and 'initrd ' are just used by 'boot.ipxe'. Now need these uris without prefix for 'boot.libvirt', so remove them. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9e48..9f03a04 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -432,7 +432,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end @@ -496,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: add a new job field 'kernel_base_params'
by Ren Wen 08 Jan '21

08 Jan '21

Kernel params like 'initrd=xxx' is just for 'boot.ipxe'. Now need kernel params without 'initrd=xxx' for 'boot.libvirt', so add a job field 'kernel_base_params' for this. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 8 +------- src/scheduler/kernel_params.cr | 15 ++++++++++----- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..e7b9e48 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -489,13 +489,7 @@ class Job end private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + return get_initrds().map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..2319a9a 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -32,7 +31,13 @@ class Job private def set_kernel_params kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) + # *kernel_base_params* is for boot.libvirt + @hash["kernel_base_params"] = JSON.parse(kernel_params_values.to_json) + + _initrds_basenames = [] of String + _initrds_basenames = initrds_basename() unless os_mount == "container" + kernel_params_values += _initrds_basenames + # *kernel_params* is for boot.ipxe @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) end - end -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-riscv64/z9.289591 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-x86_64/z9.289594 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-aarch64/z9.289571 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..363cf4c 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v3 compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..c9b5df4 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "$compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..6174916 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

2 1