- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 08 Jan '21

08 Jan '21

daily timing work: reset email rate limit queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..62f27ea --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 08 Jan '21

08 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v5 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v4 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

2 2

review star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our review star, thanks Jihui! jimmy@crystal ~/todo/bin% ./review-star 7 Review email records in the last 7 days total review paragraph point people 53 31 68 26 Wu Fengguang 87 15 18 10 Wei Jihui 37 10 20 9 Lu Kaiyi 32 4 11 8 Yu Chuan 58 15 33 7 Luan Shengde 24 5 8 6 Xiao Shenwei 20 11 12 6 Xu Xijian 10 5 11 5 Wu Zhende 32 3 3 4 Hu Xuejiao 19 8 5 4 Liu Shaofei 22 9 9 4 Ren Wen 28 2 2 3 Cao Xueliang 29 6 5 2 Cui Yili 10 8 8 2 Wang Yong 17 2 3 2 Li Ping 3 3 5 2 Zhang Dewan 12 4 7 1 Lin Jiaxin 23 4 6 1 Wang Chenglong 10 2 2 1 Li Yuanchao 13 1 1 1 Lu Weitao 5 1 1 1 Zhang Yale 33 2 2 1 Liu Yinsi 6 1 2 0 Bai Jing

1 0

git count
by Wu Fengguang 08 Jan '21

08 Jan '21

Yu Chuan is our code star again, thanks Yu Chuan! 10 Yu Chuan 6 Wei Jihui 6 Wu Fengguang 6 Li Yuanchao 5 Liu Shaofei 4 Xu Xijian 4 Hu Xuejiao 4 Lu Weitao 4 Wu Zhende 4 Xiao Shenwei 4 Lu Kaiyi 3 Wang Chenglong 3 Li Ping 2 Bai Jing 2 Cao Xueliang 2 Cui Yili 2 Lin Jiaxin 1 Zhang Yale 1 Luan Shengde 0 Tong Qunfeng 0 Du Kaitian 0 Liu Yinsi 0 Sun Yukui 0 Zhang Yu 0 Wang Yong 0 Ren Wen 0 Zhang Dewan 0 Shi Zhichao

1 0

help star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our help star, thanks Jihui! 15.2 Wu Fengguang 13.0 Wei Jihui 12.1 Yu Chuan 11.6 Xiao Shenwei 9.6 Lu Weitao 8.3 Ren Wen 7.5 Cao Xueliang 7.3 Wu Zhende 6.8 Zhang Yale 6.6 Wang Yong 6.1 Zhang Dewan 6.0 Liu Shaofei 4.8 Liu Yinsi 4.3 Xu Xijian 3.6 Wang Chenglong 3.3 Du Kaitian 2.8 Li Ping 2.7 Lin Jiaxin 2.3 Bai Jing 2.2 Cui Yili 2.0 Luan Shengde 1.3 Hu Xuejiao 0.7 Lu Kaiyi 0.0 Tong Qunfeng 0.0 Li Yuanchao 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH lkp-tests] adaptation/openeuler: add dependency of pgbench
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/openeuler | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 09120489..1938a8aa 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -295,3 +295,4 @@ ruby-full: ruby ruby-devel rubygem-rdoc linux-base: openssh-server: openssh openssh-client: openssh +postgresql-server: postgresql -- 2.23.0

2 1

[PATCH lkp-tests] tests/ebizzy: modify the value of iterations
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/ebizzy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ebizzy b/tests/ebizzy index dcc301871..394bc09ba 100755 --- a/tests/ebizzy +++ b/tests/ebizzy @@ -19,7 +19,7 @@ duration=${duration%%s} [ -n "$mmap" ] && opt_mmap=" -m " [ -n "$linear" ] && opt_linear=" -z " - +[[ "$iterations" =~ ^[0-9]{3}x$ ]] && iterations=100 for i in $(seq 1 $iterations) do echo Iteration: $i -- 2.23.0

3 2

[PATCH v2 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 08 Jan '21

08 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 08 Jan '21

08 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..cfc70ec --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|logo', '--name name|email|logo', 'appoint a name|email|logo to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || nil + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH compass-ci] uniform constant naming
by Lu Weitao 08 Jan '21

08 Jan '21

[Why] most file use "SEND_MAIL_HOST/PORT" [How] sed -i "s/[^_]MAIL_HOST/SEND_MAIL_HOST/g" `grep MAIL_HOST -rl compass-ci/*` sed -i "s/[^_]MAIL_PORT/SEND_MAIL_PORT/g" `grep MAIL_PORT -rl compass-ci/*` Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/delimiter/start | 4 ++-- container/extract-stats/start | 4 ++-- lib/constants.rb | 4 ++-- lib/mail_client.rb | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index 5d7037d..cb44d84 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -9,8 +9,8 @@ require_relative '../defconfig.rb' names = Set.new %w[ ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT MONITOR_HOST MONITOR_PORT TASKQUEUE_HOST diff --git a/container/extract-stats/start b/container/extract-stats/start index 74cf917..6510564 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,8 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT LAB ] diff --git a/lib/constants.rb b/lib/constants.rb index c3c7543..a274adc 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +SEND_MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 diff --git a/lib/mail_client.rb b/lib/mail_client.rb index f63e65d..6f5a099 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -8,8 +8,8 @@ require_relative 'constants.rb' # mail client class class MailClient - HOST = (ENV.key?('MAIL_HOST') ? ENV['MAIL_HOST'] : MAIL_HOST) - PORT = (ENV.key?('MAIL_PORT') ? ENV['MAIL_PORT'] : MAIL_PORT).to_i + HOST = (ENV.key?('SEND_MAIL_HOST') ? ENV['SEND_MAIL_HOST'] : SEND_MAIL_HOST) + PORT = (ENV.key?('SEND_MAIL_PORT') ? ENV['SEND_MAIL_PORT'] : SEND_MAIL_PORT).to_i def initialize(host = HOST, port = PORT) @host = host @port = port -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/auto_submit.yaml: add/update linux repos
by Lin Jiaxin 08 Jan '21

08 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit.yaml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 69b73ff..22172eb 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,7 +4,19 @@ c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml l/linux/linux: -- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/linux-next: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/openeuler-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/debian-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/archlinux-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

3 2

[PATCH lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 08 Jan '21

08 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [ 75.489958] Kernel tests: Boot OK! ... [ 82.149060] sysrq: Resetting [why] check network ok or not by checking network adapter is not accurate, use ping ip more accurate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3bd..b7795b40a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER | grep -q ttl && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 08 Jan '21

08 Jan '21

Prefix 'kernel ' and 'initrd ' are just used by 'boot.ipxe'. Now need these uris without prefix for 'boot.libvirt', so remove them. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9e48..9f03a04 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -432,7 +432,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end @@ -496,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: add a new job field 'kernel_base_params'
by Ren Wen 08 Jan '21

08 Jan '21

Kernel params like 'initrd=xxx' is just for 'boot.ipxe'. Now need kernel params without 'initrd=xxx' for 'boot.libvirt', so add a job field 'kernel_base_params' for this. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 8 +------- src/scheduler/kernel_params.cr | 15 ++++++++++----- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..e7b9e48 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -489,13 +489,7 @@ class Job end private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + return get_initrds().map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..2319a9a 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -32,7 +31,13 @@ class Job private def set_kernel_params kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) + # *kernel_base_params* is for boot.libvirt + @hash["kernel_base_params"] = JSON.parse(kernel_params_values.to_json) + + _initrds_basenames = [] of String + _initrds_basenames = initrds_basename() unless os_mount == "container" + kernel_params_values += _initrds_basenames + # *kernel_params* is for boot.ipxe @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) end - end -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-riscv64/z9.289591 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-x86_64/z9.289594 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-aarch64/z9.289571 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..363cf4c 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v3 compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..c9b5df4 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "$compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..6174916 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] assign-account: add 'lab' to default yaml
by Luan Shengde 08 Jan '21

08 Jan '21

the default yaml file will contain: my_email my_name lab lab is used to read the lab yaml according to its value. example: if the value of lab is 'z9', the sched will read token from file: z9.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lib/job.rb: add fault_reproduction
by Wei Jihui 08 Jan '21

08 Jan '21

do fault reproduction when submit a failed job Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8346db87..631ebfc9 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -255,6 +255,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) + hash.merge!(fault_reproduction) if hash.has_key?("id") @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -285,6 +286,10 @@ class Job @jobfile = jobfile end + def fault_reproduction + wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + end + def multi_args(hash) jobs_array = [] hash.each { |key, value| -- 2.23.0

2 1

[PATCH lkp-tests] bootstrap: don't try to install ntpdate
by Wu Fengguang 08 Jan '21

08 Jan '21

It's not necessary for every boot. Let's not slow down every test run. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/bootstrap.sh | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3b..b04ea3dd 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -237,20 +237,6 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - has_cmd ntpdate || { - detect_system - local DISTRO=$_system_name_lowercase - [ "$DISTRO" == "unknown" ] && { - echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." - return - } - has_cmd ruby || { - echo "[WARNING] Cannot find ruby in $DISTRO, so skip run_ntpdate." - return - } - . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) - } - has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] The `run-ipconfig` step is the IP configuration step for initramfs; The `[ -z "$net_devices"]` step is check whether the network is normal. Now the `check normal` step is before 'ip configuration' step, so we encounter the following error: + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return [Debug Msg] + [ ! -e /usr/share/initramfs-tools/scripts/functions ] + get_net_devices + local net_devices + local i + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ p0s1/ != /sys/class/net/enp0s1/ ] + basename /sys/class/net/enp0s1/ + net_devices= enp0s1 + [ != /sys/class/net/lo/ ] + continue + echo enp0s1 + local net_devices= enp0s1 + [ -z ] + warn_no_eth0 + [ -f /proc/config.gz ] + return + echo ls /sys/class/net + ls /sys/class/net + is_virt + [ -n ] + [ -n ] + has_cmd virt-what + command -v virt-what + grep -q -w hypervisor /proc/cpuinfo + return 0 + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

2 3

[PATCH v3 lkp-tests] lib/bootstrap.sh: fix error when os is not integrated into lkp-tests
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] If the current os has not been integrated into lkp-tests, run method `detect_system`, then var `$_system_name_lowercase` will be 'unknown'. So we should adapt this. [Error] + local DISTRO=unknown ++ echo ntpdate ++ /home/yuchuan/lkp-tests/sbin/adapt-packages unknown Traceback (most recent call last): 3: from /home/yuchuan/lkp-tests/sbin/adapt-packages:11:in `<main>' 2: from /usr/share/ruby/psych.rb:497:in `load_file' 1: from /usr/share/ruby/psych.rb:497:in `open' /usr/share/ruby/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /home/yuchuan/lkp-tests/distro/adaptation/unknown (Errno::ENOENT) [How] If the current os has not been integrated into lkp-tests, then return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..912a76bb6687 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -240,6 +240,10 @@ run_ntpdate() has_cmd ntpdate || { detect_system local DISTRO=$_system_name_lowercase + [ "$DISTRO" == "unknown" ] && { + echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 3

[PATCH v5 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-riscv64/z9.288315 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-x86_64/z9.288332 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-aarch64/z9.288335 [how] put public options like $kernel, $initrd, $smp together, list different options like $machine, $bios, $nic according to different qemu command, then merge all options. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 110 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 95 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..a8a2b33 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,56 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +121,78 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) + ;; + esac + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option run_qemu -- 2.23.0

3 4