- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] taskqueue: add build dependent container: redis
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] The container taskqueue depends on redis, so add redis to taskqueue/build-depends. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/taskqueue/build-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/build-depends b/container/taskqueue/build-depends index 5dd403e166af..a961b878fb44 100755 --- a/container/taskqueue/build-depends +++ b/container/taskqueue/build-depends @@ -1 +1,2 @@ scheduler-dev +redis -- 2.23.0

2 4

[PATCH v4 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 23 Jan '21

23 Jan '21

request url: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

2 1

[PATCH v4 compass-ci 2/3] libvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in this way, user can get message when error occurred [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin/quick_fetch: add an interface script for git_mirror speed up
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/quick_fetch | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 sbin/quick_fetch diff --git a/sbin/quick_fetch b/sbin/quick_fetch new file mode 100755 index 0000000..e8378ec --- /dev/null +++ b/sbin/quick_fetch @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +# usage: quick_fetch git_repo:$git_repo +# or: quick_fetch $url +# quick_fetch https://github.com/Siguyi/AvxToNeon +# quick_fetch git_repo:u/upstream-repos/upstream-repos +def send_url_of_repo + connection = Bunny.new('amqp://172.17.0.1:5672') + connection.start + channel = connection.create_channel + + queue = channel.queue('web_hook') + message = ARGV[0] + queue.publish(message) + connection.close +end + +send_url_of_repo -- 2.23.0

1 0

[PATCH compass-ci 1/2] git_mirror.rb: add one more parameter choice when use webhook
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6d1624f..a111e51 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -318,6 +318,8 @@ class MirrorMain # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) + return webhook_url.split(':')[1].gsub(' ', '') if webhook_url =~ /^(git_repo:)/ + fork_name, project = webhook_url.split('/')[-2, 2] git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] bvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in the fulture [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 in this way, user can perceived message when error occurred Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 2

[PATCH compass-ci] sparrow/README.md: improve readability.
by Li Ping 23 Jan '21

23 Jan '21

[why] Markdown does not translate every line break character in a paragraph into a <br/> tag. ref: http://markdown.cn/ obviously, in some paragraphs of this markdown document, the newline effect is not shown, content would be displayed as a single line, readability would be reduced. --- sparrow/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/README.md b/sparrow/README.md index c7450fd..1906f10 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -72,8 +72,10 @@ ``` 执行上述命令正常情况下会提示信息如下: + ``` submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 23 Jan '21

23 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 23 Jan '21

23 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: reformatting the document
by Li Ping 23 Jan '21

23 Jan '21

[why] In markdown documents, the newline effect is not shown, content not placed in ``` ''' would be displayed as a single line, readability would be reduced. Signed-off-by: Li Ping <1477412247(a)qq.com> --- sparrow/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/README.md b/sparrow/README.md index 794e433..3d7cdd0 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -72,8 +72,10 @@ ``` 执行上述命令正常情况下会提示信息如下: + ``` submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） -- 2.23.0

3 2

[PATCH v2 lkp-tests 3/3] distro/depends: add nginx-server-dev
by Wei Jihui 23 Jan '21

23 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server-dev | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/nginx-server-dev diff --git a/distro/depends/nginx-server-dev b/distro/depends/nginx-server-dev new file mode 100644 index 00000000..170af747 --- /dev/null +++ b/distro/depends/nginx-server-dev @@ -0,0 +1,2 @@ +pcre-devel +libpcre3-dev -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] pkg/nginx-server: add ngixn conf
by Wei Jihui 23 Jan '21

23 Jan '21

nginx-server test need forbidden nf_conntrack when os setup Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx-server/PKGBUILD | 10 ++++++++-- pkg/nginx-server/blacklist.conf | 10 ++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) create mode 100644 pkg/nginx-server/blacklist.conf diff --git a/pkg/nginx-server/PKGBUILD b/pkg/nginx-server/PKGBUILD index 141722e1..ac458776 100644 --- a/pkg/nginx-server/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -5,8 +5,8 @@ arch=('i386' 'x86_64') url="https://nginx.org" license=('BSD') nginx_name=("nginx-${pkgver}.${pkgrel}") -source=("https://nginx.org/download/${nginx_name}.tar.gz") -md5sums=('SKIP') +source=("https://nginx.org/download/${nginx_name}.tar.gz" "blacklist.conf") +md5sums=('SKIP' 'SKIP') build() { @@ -19,6 +19,12 @@ build() package() { + mkdir -p "$pkgdir/rootfs/addon/etc/modprobe.d/" + cp blacklist.conf "$pkgdir/rootfs/addon/etc/modprobe.d/" + + mkdir -p "$pkgdir/lkp/benchmarks/${pkgname}/" + cp -r "$LKP_SRC/pkg/${pkgname}/nginx_conf" "$pkgdir/lkp/benchmarks/${pkgname}/" + cd "${srcdir}/${nginx_name}" make install DESTDIR="${pkgdir}" } diff --git a/pkg/nginx-server/blacklist.conf b/pkg/nginx-server/blacklist.conf new file mode 100644 index 00000000..e8798cc7 --- /dev/null +++ b/pkg/nginx-server/blacklist.conf @@ -0,0 +1,10 @@ +install nf_conntrack /bin/false +blacklist nf_conntrack +blacklist nf_conntrack_ipv6 +blacklist xt_conntrack +blacklist nf_conntrack_ftp +blacklist xt_state +blacklist iptable_nat +blacklist ipt_REDIRECT +blacklist nf_nat +blacklist nf_conntrack_ipv4 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] pkg: rename nginx to nginx-server
by Wei Jihui 23 Jan '21

23 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/{nginx => nginx-server}/PKGBUILD | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) rename pkg/{nginx => nginx-server}/PKGBUILD (63%) diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx-server/PKGBUILD similarity index 63% rename from pkg/nginx/PKGBUILD rename to pkg/nginx-server/PKGBUILD index 989676c4..141722e1 100644 --- a/pkg/nginx/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -1,23 +1,24 @@ -pkgname=nginx +pkgname=nginx-server pkgver=1.14 pkgrel=2 arch=('i386' 'x86_64') url="https://nginx.org" license=('BSD') -source=("https://nginx.org/download/nginx-${pkgver}.${pkgrel}.tar.gz") +nginx_name=("nginx-${pkgver}.${pkgrel}") +source=("https://nginx.org/download/${nginx_name}.tar.gz") md5sums=('SKIP') build() { benchmark_path="/lkp/benchmarks/${pkgname}" mkdir -p "${benchmark_path}" - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/${nginx_name}" ./configure --with-http_ssl_module --prefix="${benchmark_path}" make -j4 } package() { - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/${nginx_name}" make install DESTDIR="${pkgdir}" } -- 2.23.0

1 0

[PATCH lkp-tests] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 22 Jan '21

22 Jan '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 1

[PATCH compass-ci] sparrow/1-storage: remove mount os rootfs
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] for locally deploy Compass CI with one click, we open directory /srv/os to download os rootfs, rather than open shared directory to mount. so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 3 --- 1 file changed, 3 deletions(-) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index eb7270d..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,3 @@ dirs=( ) mkdir -p "${dirs[@]}" - -mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr -mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/deploy-cci: mount os rootfs
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] mount our server's os rootfs directory is a dedicated way for $LKP_SRC/test/deploy-cci, it is not a general way for deploy Compass CI with one click, put it in tests/deploy-cci more reasonable, not in $CCI_SRC/sparrow script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index 8e6acbac4..e2a7bd320 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -35,5 +35,13 @@ deploy() ./install-tiny } +mount_rootfs() +{ + mkdir -p /srv/{os,initrd} + mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +} + export_env +mount_rootfs deploy -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/depends: fix failed to mount
by Liu Yinsi 22 Jan '21

22 Jan '21

[error] root@test-lys /c/compass-ci/container/os-cifs# mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr mount: /srv: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program. [why] missing depends cifs-utils causes mount error Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- distro/depends/deploy-cci | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/deploy-cci b/distro/depends/deploy-cci index 5664e303b..c0d6a5310 100644 --- a/distro/depends/deploy-cci +++ b/distro/depends/deploy-cci @@ -1 +1,2 @@ git +cifs-utils -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_error_messages.rb: return false when matching error filename failed
by Lin Jiaxin 22 Jan '21

22 Jan '21

If error_id = 'build-pkg.c++:fatal-error:switch'~x16~xe6~xab~xaa'does-not-start-with'-'', cannot get the error filename and return an empty hash. The empty hash is not credible too. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 2e35019..5f74be7 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -17,7 +17,9 @@ def credible?(previous_job_id, later_job_id, error_id) previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') - return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + filenames_check = filenames_check(previous_result_file, later_result_file, error_id) + + return false if filenames_check.empty? || filenames_check.value?(false) return true end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare_error_messages.rb: remove filename extension when matching error filename
by Lin Jiaxin 22 Jan '21

22 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 92533b8..2e35019 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -28,8 +28,8 @@ def filenames_check(previous_result_file, later_result_file, error_id) error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) error_lines.each do |error_line| - # "src/ssl_sock.c:1454:104: warning: unused parameter 'al' [-Wunused-parameter]" => "src/ssl_sock.c" - filenames << $1 if error_line =~ /(.*)(:\d+){2}: (error|warning):/ + # "src/ssl_sock.c:1454:104: warning: unused parameter 'al' [-Wunused-parameter]" => "src/ssl_sock" + filenames << $1.chomp(File.extname($1)) if error_line =~ /(.*)(:\d+){2}: (error|warning):/ end File.open(later_result_file).each_line do |line| -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 22 Jan '21

22 Jan '21

add limit for borrowing machine: runtime <= 10 days the job will break up if the value of runtime in the job yaml is greater than 10 days Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..bb9b792 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -64,6 +64,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + check_run_time() set_sshr_info() return end @@ -410,6 +411,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash["testcase"] == "borrow" + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/dump-stats: support dump 'message'
by Lu Weitao 22 Jan '21

22 Jan '21

[Example] { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ], "iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/dump-stat | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/dump-stat b/sbin/dump-stat index b5abf5a9b..6ebda1f29 100755 --- a/sbin/dump-stat +++ b/sbin/dump-stat @@ -61,10 +61,8 @@ while (line = STDIN.gets) end # only number is valid unless v.numeric? - invalid_records.push record_index - warn_stat "invalid stats value: #{v}", monitor - - exit 1 + result[k] = v + next end v = v.index('.') ? v.to_f : v.to_i @@ -84,6 +82,8 @@ min_cols_stat = '' max_cols_stat = '' zero_stats = [] result.each do |key, val| + next if val.is_a?(String) + if max_cols < val.size max_cols = val.size max_cols_stat = key -- 2.23.0

1 1

[PATCH v2 compass-ci 2/3] libvirt/lib: introduce mylog.rb
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in the fulture [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 in this way, user can perceived message when error occurred Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..141dd80 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + @name = filename + rm_exist? + super(filename) + format + end + + private + + def rm_exist? + if FileTest.exist?(@name) + File.delete(@name) + end + end + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: set default values
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] when call simplify-ci individually to deploy environment, SCHED_HOST, SCHED_PORT, GIT_SERVER, these variables are undefined in a very clean machine, execute the script will failed, so set default values is necessary. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f772..df18272f3 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,18 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${SCHED_HOST:=172.17.0.1} +: ${SCHED_PORT:=3000} + git_ci() { + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 22 Jan '21

22 Jan '21

the maximum time for borrowing machine should no more than 10 days. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..e9e693c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -64,6 +64,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + check_run_time() set_sshr_info() return end @@ -410,6 +411,19 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash["testcase"] == "borrow" + + if @hash["runtime"].ends_with? "d" + hour = hash["runtime"][0..-2].to_i * 24 + elsif @hash["runtime"].ends_with? "h" + hour = hash["runtime"][0..-2].to_i + end + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hour.is_a?(Int32) && hour >= 240 + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb support merge 'message' to stats
by Lu Weitao 22 Jan '21

22 Jan '21

[Example] { "iperf.tcp.sender.bps": 34804801216.197174, "iperf.tcp.receiver.bps": 34804762215.18231, "iperf.iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..5920d62 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -10,6 +10,8 @@ require "#{LKP_SRC}/lib/matrix" require_relative './params_group.rb' def set_pre_value(item, value, sample_size) + return value if value.is_a?(String) + if value.size == 1 value[0] elsif independent_counter? item -- 2.23.0

1 0

[PATCH compass-ci] lib/sched.cr: reset testbox's deadline
by Wu Zhende 22 Jan '21

22 Jan '21

The life cycle of the testbox is from booting to rebooting. "deadline" is set during booting and reset during rebooting. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 8d7c218..f46d97c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -104,7 +104,7 @@ class Sched def update_tbox_wtmp testbox = "" - hash = Hash(String, String).new + hash = Hash(String, String | Nil).new time = Time.local.to_s("%Y-%m-%d %H:%M:%S") hash["time"] = time @@ -116,6 +116,7 @@ class Sched testbox = value when "tbox_state" hash["state"] = value + hash["deadline"] = nil if value == "rebooting" when "mac" hash["mac"] = normalize_mac(value) else -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: optimize borrow-machine.zh.md
by Luan Shengde 22 Jan '21

22 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.zh.md | 308 ++++++++------------------------ 1 file changed, 79 insertions(+), 229 deletions(-) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 916aea7..73aea6b 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -1,243 +1,93 @@ -# 前提条件 - -请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… -- send apply account email. -- receive email from compass-ci-robot(a)qq.com. -- local environment configuration. - - -# 申请测试机（虚拟机） - -1. 生成本地RSA公私钥对 - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. 根据需求选择yaml - - 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. 提交yaml并连接测试机（虚拟机） - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 - -4. 使用完毕退还测试机（虚拟机） - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - - -# 申请测试机（物理机） - - -1. 生成本地RSA公私钥对 - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. 根据需求选择yaml - - 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. 提交yaml并连接测试机（物理机） - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 +# 前置准备 +- 申请account +- 配置默认yaml文件 - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 +如果未完成以上步骤，请参考[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)完成操作。 - System information as of time: Thu Nov 26 14:51:59 CST 2020 +# 申请测试机 - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 - -4. 使用完毕退还测试机（物理机） - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +## 1. 生成本地公钥 + 使用下面命令查看是否已存在ssh公钥： + + ls ~/.ssh/*.pub + + 如果当前没有现成的公钥，请使用下面命令进行生成： + + ssh-keygen + +## 2. 选择job yaml + + 在 ~/lkp-tests/jobs 目录下为您提供了众多job实例。 + + 使用以下命令可筛选出借用机器的实例： + + cd ~/lkp-tests/jobs + ls borrow* + +## 3. 提交job + + 使用以下命令样例提交job： + + 容器: + + submit -c -m testbox=dc-2g os_mount=container docker_image=centos:8 borrow-1h.yaml + + 虚拟机: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + 物理机： + + submit -c -m testbox=taishan200-2280-2s48p-256g borrow-1h.yaml + + - 使用以上命令，您可以实时查看job状态，测试机正常运行后将直接登入。 + - 借用成功后，会同时发送邮件通知，提供测试机配置信息及登录命令，请注意查收。 + - 租借期内，您可以通过邮件内提供的登录命令再次登录测试机。 + - 登录命令仅对当前测试机有效，测试机归还后，登录命令将不可用。 + +## 4. 退还测试机 + + 手动归还（推介使用）： + + 测试机使用完成后，及时执行‘reboot’命令进行测试机归还，避免闲置造成资源浪费。 + + 到期自动归还： + + 借用期限到达后，测试接将自动重启归还。 + + - 所有测试机在执行‘reboot’命令后都会被归还，归还后不可再次登录使用。 + - 机器归还后，如果您还需要继续使用测试机，请重新提交job申请新的测试机。 # FAQ +## 自定义借用时长 + + 在借用机器的yaml实例文件中，找到’runtime‘字段，根据需求修改借用时长。 + + - 借用机器可以按照小时（h）、天（d）来计算。 + - 借用时长最多不超过10天。 + +## submit命令指导 + + 学习submit命令，您可以使用以下命令查看submit命令的各项参数及使用方法： + + submit -h + + 也可以参考submit命令手册学习submit命令高级用法： + + [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + +## 可选的testbox -* 如何自行修改申请时长 - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # 使用vim来修改你的runtime字段的值 - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # 修改完毕后重新submit即可 - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Submit命令指导 - - 参考文档：[submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - -* testbox有什么可选项 - - testbox可选项请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + 查看可选的testbox列表，请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts >![](./../public_sys-resources/icon-note.gif) **说明：** > - > 虚拟机的testbox : vm-xxx - > 物理机的testbox : taishan200-2280-xxx + > - 容器: dc-xxx + > - 虚拟机: vm-xxx + > - 物理机: taishan200-2280-xxx @@ -245,6 +95,6 @@ > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 -* 如何 borrow 指定的操作系统 +## 如何 borrow 指定的操作系统关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) -- 2.23.0

1 0

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 22 Jan '21

22 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. For testing ansible git repo, those cannot be tested in a unified way, It mainly tests ansible roles git repo, and these roles repo will be more and more. [why] command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. site.yml like as follow: ``` site.yml --- - hosts: localhost roles: - role: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..38ef0430 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,73 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 1

[PATCH lkp-tests 2/2] pkg/nginx-server: add blacklist.conf
by Wei Jihui 22 Jan '21

22 Jan '21

nginx-server test need forbidden nf_conntrack when os setup Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx-server/PKGBUILD | 3 +++ pkg/nginx-server/blacklist.conf | 10 ++++++++++ 2 files changed, 13 insertions(+) create mode 100644 pkg/nginx-server/blacklist.conf diff --git a/pkg/nginx-server/PKGBUILD b/pkg/nginx-server/PKGBUILD index a8a7091b..95810b1a 100644 --- a/pkg/nginx-server/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -20,4 +20,7 @@ package() { cd "${srcdir}/nginx-${pkgver}.${pkgrel}" make install DESTDIR="${pkgdir}" + + mkdir -p "$pkgdir/rootfs/addon/etc/modprobe.d/" + cp "$LKP_SRC/pkg/${pkgname}/blacklist.conf" "$pkgdir/rootfs/addon/etc/modprobe.d/" } diff --git a/pkg/nginx-server/blacklist.conf b/pkg/nginx-server/blacklist.conf new file mode 100644 index 00000000..e8798cc7 --- /dev/null +++ b/pkg/nginx-server/blacklist.conf @@ -0,0 +1,10 @@ +install nf_conntrack /bin/false +blacklist nf_conntrack +blacklist nf_conntrack_ipv6 +blacklist xt_conntrack +blacklist nf_conntrack_ftp +blacklist xt_state +blacklist iptable_nat +blacklist ipt_REDIRECT +blacklist nf_nat +blacklist nf_conntrack_ipv4 -- 2.23.0

1 0

[PATCH lkp-tests 1/2] pkg: rename nginx to nginx-server
by Wei Jihui 22 Jan '21

22 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/{nginx => nginx-server}/PKGBUILD | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename pkg/{nginx => nginx-server}/PKGBUILD (78%) diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx-server/PKGBUILD similarity index 78% rename from pkg/nginx/PKGBUILD rename to pkg/nginx-server/PKGBUILD index 989676c4..a8a7091b 100644 --- a/pkg/nginx/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -1,4 +1,4 @@ -pkgname=nginx +pkgname=nginx-server pkgver=1.14 pkgrel=2 arch=('i386' 'x86_64') @@ -11,13 +11,13 @@ build() { benchmark_path="/lkp/benchmarks/${pkgname}" mkdir -p "${benchmark_path}" - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/nginx-${pkgver}.${pkgrel}" ./configure --with-http_ssl_module --prefix="${benchmark_path}" make -j4 } package() { - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/nginx-${pkgver}.${pkgrel}" make install DESTDIR="${pkgdir}" } -- 2.23.0

1 0

[PATCH compass-ci] doc: fix format and spell error
by Liu Yinsi 22 Jan '21

22 Jan '21

1. add 2 space fix no newline 2. golbal -> global, verison -> version Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 48 +++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 794e433..c7450fd 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -3,24 +3,24 @@ 在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 ## 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) - >**说明：** - >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 + >**说明：** + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 - >**说明：** + >**说明：** >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -31,8 +31,8 @@ 2. 配置git账号 ```bash - git config --golbal user.name "XXX" - git config --golbal user.email "xxx(a)aa.com" + git config --global user.name "xxx" + git config --global user.email "xxx(a)xxx.com" ``` 3. 设置文件权限并关闭SELINUX @@ -40,8 +40,8 @@ umask 002 && setenforce 0 ``` - >**说明：** - >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 + >**说明：** + >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 4. 创建工作目录并克隆 compass-ci 项目代码 @@ -71,9 +71,9 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下: - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） @@ -107,7 +107,7 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… ```bash cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 cat output -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: if qemu not exist, output and exit
by Xu Xijian 22 Jan '21

22 Jan '21

If there is no qemu in compass-ci's environment, output something to remind users and then exit. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..70d3ca6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -193,6 +193,10 @@ individual_option() -netdev $netdev ) ;; + *) + echo "qemu not found: $qemu" + exit + ;; esac } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 22 Jan '21

22 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 22 Jan '21

22 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] libvirt based on xml file to run vm, so we need parse xml file nokogiri is a excellent tool of parse xml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH v6 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 46 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..7f64faee --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,46 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +main() +{ + file_check + tbox_check + reset_tbox || { + echo "Failed to reboot the testbox: $host_name" + exit 2 + } +} + +main -- 2.23.0

1 0

[PATCH v5 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 +++++++++ tests/reset-tbox | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..ae3cbe52 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,52 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +result_check() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + file_check + tbox_check + reset_tbox + result_check +} + +main -- 2.23.0

2 2

[PATCH compass-ci] sparrow/1-storage: fix failed to set permission
by Liu Yinsi 22 Jan '21

22 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 ~/compass-ci/sparrow/1-storage# ./permission chown: changing ownership of '/srv/initrd': Read-only file system [why] mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr because mount permission for /srv/initrd is read-only, cause set permission failed. [how] for user install compass-ci at their local server, no need to mount directory, they will download rootfs by wget link, so remove it. mount rootfs can move to selftest own tests deploy-cci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 3 --- 1 file changed, 3 deletions(-) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index eb7270d..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,3 @@ dirs=( ) mkdir -p "${dirs[@]}" - -mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr -mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

2 2

[PATCH compass-ci] add a comment, make it easier to understand
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- providers/qemu/kvm.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..9034ccf 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -89,6 +89,7 @@ set_initrd() set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + # when arch='x86_64', use the follwing file: [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd } -- 2.23.0

2 4

[PATCH compass-ci] doc/manual: optimize submit-job.zh.md
by Wu Zhende 22 Jan '21

22 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- doc/manual/submit-job.zh.md | 34 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 22 deletions(-) diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 540d91f..92faa5c 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -8,32 +8,22 @@ submit 命令的作用是提交测试任务。该命令提供了多个选项帮助用户更灵活的提交任务，您可以在命令行直接输入 summit 命令来查看帮助信息，并根据实际需求灵活使用。 - ### 基本用法 -测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。使用如下命令提交测试任务： - -``` -submit iperf.yaml -``` -如下所示： - -```shell -hi8109@account-vm ~% submit iperf.yaml -submit iperf.yaml, got job_id=z9.173924 -``` +测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。 -本文所示的 yaml 文件均已添加 testbox 字段，若您提交的 yaml 文件不包含该字段将报错： +若您的yaml文件不包含testbox字段，直接提交将会报错： ```shell hi8109@account-vm ~% submit iperf.yaml submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' ``` -您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： +因为testbox是必填字段，您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： -``` -submit iperf.yaml testbox=vm-2p8g +```shell +hi8109@account-vm ~% submit iperf.yaml testbox=vm-2p8g +submit iperf.yaml, got job_id=z9.173924 ``` testbox 字段的值指定需要的测试机，可以使用 `ll` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下所示： @@ -115,7 +105,7 @@ options: 使用-o DIR 命令可以将最终生成的yaml文件保存到指定目录 DIR 下。示例如下所示： ``` - submit -o ~/iperf.yaml + submit -o ~/iperf.yaml testbox=vm-2p8g ``` 运行命令之后会在指定目录生成经过 submit 处理过的 yaml 文件。 @@ -126,7 +116,7 @@ options: 示例命令如下： ``` - submit -a iperf.yaml + submit -a iperf.yaml testbox=vm-2p8g ``` ``` @@ -141,13 +131,13 @@ options: 示例命令如下： ``` - submit -m iperf.yaml + submit -m iperf.yaml testbox=vm-2p8g ``` 控制台显示如下： ```shell - hi8109@account-vm ~% submit -m iperf.yaml + hi8109@account-vm ~% submit -m iperf.yaml testbox=vm-2p8g submit iperf.yaml, got job_id=z9.173923 query=>{"job_id":["z9.173923"]} connect to ws://172.168.131.2:11310/filter @@ -173,14 +163,14 @@ options: 示例命令如下： ``` - submit -m -c borrow-1h.yaml + submit -m -c borrow-1h.yaml testbox=vm-2p8g ``` 当我们提交一个申请设备的任务后，会获取到返回的登陆信息，如 `ssh ip -p port`，添加 -c 参数之后不需要我们手动输入 ssh 登陆命令来进入执行机。控制台显示如下： ```shell - hi8109@account-vm ~% submit -m -c borrow-1h.yaml + hi8109@account-vm ~% submit -m -c borrow-1h.yaml testbox=vm-2p8g submit borrow-1h.yaml, got job_id=z9.173925 query=>{"job_id":["z9.173925"]} connect to ws://172.168.131.2:11310/filter -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: use an independent interface for submit
by Wu Zhende 22 Jan '21

22 Jan '21

When we use Job.new, we need to do some extra processing only when we submit the job. Use interface "initialized?" to determine whether to submit a job. However, there is a vulnerability. If a job with all fields initialized is submitted, the job can be successfully submitted without checking the account. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 29 ++++++++++------------------- src/scheduler/submit_job.cr | 1 + 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..964c520 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -55,25 +55,6 @@ class Job @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new - - # init job with "-1", or use the original job_content["id"] - id = "-1" if "#{id}" == "" - - if initialized? - if @hash["id"] == "#{id}" - return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") - - check_account_info() - set_sshr_info() - return - end - end - - @hash["id"] = JSON::Any.new("#{id}") - - check_required_keys() - check_account_info() - set_defaults() end METHOD_KEYS = %w( @@ -155,6 +136,16 @@ class Job end end + def submit(id = nil) + # init job with "-1", or use the original job_content["id"] + id = "-1" if "#{id}" == "" + @hash["id"] = JSON::Any.new("#{id}") + + check_required_keys() + check_account_info() + set_defaults() + end + private def set_defaults append_init_field() set_docker_os() diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index bcb265f..1837cb1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -7,6 +7,7 @@ class Sched job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) + job.submit(job_content["id"]?) job["commit_date"] = get_commit_date(job) cluster_file = job["cluster"] -- 2.23.0

1 0

[PATCH v2 compass-ci] add a comment, make it easier to understand
by Zhang Dewan 21 Jan '21

21 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- providers/qemu/kvm.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..d445248 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -89,6 +89,7 @@ set_initrd() set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + # when arch='x86_64', use the following file: [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd } -- 2.23.0

1 0

[PATCH lkp-tests] expose/report error on install_proxy()
by Wu Fengguang 21 Jan '21

21 Jan '21

Carefully preserve error code from external commands, and return error code to function callers. Check/report error, and fail it early. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- bin/lkp-setup-rootfs | 28 +++++++++++++--------------- lib/http.sh | 8 ++++++++ 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..cf48c8f6b 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -103,8 +103,6 @@ wget_install_cgz() install_proxy_yum() { - [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo @@ -115,8 +113,6 @@ install_proxy_yum() install_proxy_apt() { - [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; @@ -133,8 +129,8 @@ EOF install_docker_proxy() { - [ -n "$DOCKER_PROXY_HOST" ] || return - [ -n "$DOCKER_PROXY_PORT" ] || return + [ -n "$DOCKER_PROXY_HOST" ] || return 0 + [ -n "$DOCKER_PROXY_PORT" ] || return 0 mkdir -p /etc/systemd/system/docker.service.d cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf @@ -156,30 +152,32 @@ EOF modify_gem_source() { echo ':sources: ["http://rubygems.org"]' >> /root/.gemrc - - return 0 } install_goproxy() { - mkdir -p /root/.config/go/ + mkdir -p /root/.config/go/ || return echo "GOPROXY=https://goproxy.io,direct" >> /root/.config/go/env - - return 0 } install_proxy() { - modify_gem_source + modify_gem_source || return install_docker_proxy - install_goproxy + install_goproxy || return [ -z "$SQUID_HOST" ] && return [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum + if [ -d "/etc/yum/" ]; then + install_proxy_yum + elif [ -d "/etc/apt/" ]; then + install_proxy_apt + else + : + fi } # Save the result to @@ -251,7 +249,7 @@ chmod_root_ssh() chmod 600 /root/.ssh/id_rsa } -install_proxy +install_proxy || exit_with_job_state 'error:proxy' chmod_root_ssh while true; do diff --git a/lib/http.sh b/lib/http.sh index a0cb33fc2..523c86333 100755 --- a/lib/http.sh +++ b/lib/http.sh @@ -60,8 +60,16 @@ jobfile_append_var() http_get_cgi "cgi-bin/lkp-jobfile-append-var?$query_str" } +exit_with_job_state() +{ + local exit_code=$? + set_job_state "$1" + exit $exit_code +} + set_job_state() { + [ -n "$1" ] || return jobfile_append_var "job_state=$1" } -- 2.23.0

1 0

help star
by Wu Fengguang 21 Jan '21

21 Jan '21

Xueliang is our help star again, thanks Xueliang! 23.2 Wu Fengguang 15.9 Cao Xueliang 13.7 Yu Chuan 13.4 Wei Jihui 12.3 Xiao Shenwei 10.9 Luan Shengde 8.4 Liu Yinsi 8.4 Liu Shaofei 7.9 Lin Jiaxin 7.6 Cui Yili 7.5 Wu Zhende 6.9 Zhang Dewan 6.8 Lu Weitao 4.7 Zhang Yale 4.4 Ren Wen 4.2 Bai Jing 3.9 Du Kaitian 3.9 Xu Xijian 3.8 Li Yuanchao 3.4 Wang Yong 3.2 Hu Xuejiao 2.2 Lu Kaiyi 2.1 Li Ping 1.3 Wang Chenglong 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

review star
by Wu Fengguang 21 Jan '21

21 Jan '21

Jihui is our review star again, thanks Jihui! Review email records in the last 7 days total review paragraph point people 61 48 122 39 Wu Fengguang 96 31 48 36 Wei Jihui 93 19 24 33 Cao Xueliang 35 18 33 17 Xiao Shenwei 38 19 52 15 Luan Shengde 58 13 16 6 Ren Wen 29 5 9 5 Lin Jiaxin 41 8 14 5 Liu Yinsi 23 2 4 4 Yu Chuan 12 5 6 4 Xu Xijian 13 7 12 4 Liu Shaofei 3 3 4 4 Wang Yong 33 3 9 3 Zhang Yale 45 5 10 3 Wu Zhende 35 2 7 2 Wang Chenglong 39 2 4 2 Lu Kaiyi 34 4 3 2 Zhang Dewan 15 2 2 1 Hu Xuejiao 10 2 2 1 Bai Jing 9 1 1 1 Li Yuanchao 28 5 4 1 Cui Yili 11 0 0 0 Li Ping 8 4 9 0 Lu Weitao

1 0

git count
by Wu Fengguang 21 Jan '21

21 Jan '21

Xueliang is our code star, thanks Xueliang! 9 Cao Xueliang 8 Wu Zhende 7 Liu Yinsi 6 Wei Jihui 6 Ren Wen 4 Xu Xijian 4 Liu Shaofei 4 Zhang Yale 3 Yu Chuan 3 Lin Jiaxin 2 Bai Jing 2 Hu Xuejiao 2 Lu Weitao 2 Xiao Shenwei 2 Wang Chenglong 2 Li Yuanchao 1 Wu Fengguang 1 Wang Yong 1 Luan Shengde 1 Li Ping 1 Lu Kaiyi 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Cui Yili 0 Zhang Dewan 0 Shi Zhichao

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

On Tue, Jan 05, 2021 at 10:03:22AM +0800, Li Ping wrote: >[Usage]: >submit -m -l job.yaml > ==> lftp -c mirror http://SRV_HTTP_HOST:SRV_HTTP_PORT/RESULT_ROOT -l = lftp? "download results" can be more meaningful feature name. lftp is just one of the clients. Thanks, Fengguang

2 1

[PATCH lkp-tests] daptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

TO ALL >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' DON't use IP address. Use domain name instead. Thanks, Fengguang

2 1

[PATCH lkp-tests] adaptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/centos | 1 + distro/adaptation/debian | 1 + 2 files changed, 2 insertions(+) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index 2cfe3877..8fbae666 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -335,3 +335,4 @@ liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: openssh-client: openssh-clients +postgresql-server: postgresql diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] providers/libvirt: customizing logger format
by Xiao Shenwei 21 Jan '21

21 Jan '21

format: datetime-severity: message such as: 2021-01-20 16:04:03 +0800-INFO: xxxxxx Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..69061ca --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + @name = filename + if FileTest.exist?(filename) + File.delete(filename) + end + super(filename) + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _, msg| + "#{datetime}-#{severity}: #{msg}\n" + end + end +end -- 2.23.0

3 7

[PATCH compass-ci] lib/job.cr: use an independent interface for submit
by Wu Zhende 21 Jan '21

21 Jan '21

When we use Job.new, we need to do some extra processing only when we submit the job. Use interface "initialized?" to determine whether to submit a job. However, there is a vulnerability. If a job with all fields initialized is submitted, the job can be successfully submitted without checking the account. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 29 ++++++++++------------------- src/scheduler/submit_job.cr | 1 + 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..964c520 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -55,25 +55,6 @@ class Job @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new - - # init job with "-1", or use the original job_content["id"] - id = "-1" if "#{id}" == "" - - if initialized? - if @hash["id"] == "#{id}" - return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") - - check_account_info() - set_sshr_info() - return - end - end - - @hash["id"] = JSON::Any.new("#{id}") - - check_required_keys() - check_account_info() - set_defaults() end METHOD_KEYS = %w( @@ -155,6 +136,16 @@ class Job end end + def submit(id = nil) + # init job with "-1", or use the original job_content["id"] + id = "-1" if "#{id}" == "" + @hash["id"] = JSON::Any.new("#{id}") + + check_required_keys() + check_account_info() + set_defaults() + end + private def set_defaults append_init_field() set_docker_os() diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index bcb265f..1837cb1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -7,6 +7,7 @@ class Sched job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) + job.submit(job_content["id"]?) job["commit_date"] = get_commit_date(job) cluster_file = job["cluster"] -- 2.23.0

2 1

[PATCH v3 lkp-tests] sbin/submit: add -f option for uploading libvirt config
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt FILE's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.rb | 11 +++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc98..812f8c299 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -16,6 +16,17 @@ require 'set' require 'pp' require 'English' +# copied from compass-ci/container/defconfig.rb +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def restore(ah, copy) if ah.class == Hash ah.clear.merge!(copy) diff --git a/sbin/submit b/sbin/submit index 39cdcf576..1d56369ad 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) unless opt_upload_files.empty? + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

3 4

[PATCH v4 lkp-tests] sbin/submit: add option -i for submit
by Wei Jihui 21 Jan '21

21 Jan '21

submit -i is for add other yamls content to job.yaml [usage] case 1: submit -i 1.yaml -i 2.yaml job.yaml var priority: 1.yaml < 2.yaml < job.yaml case 2: submit -i 1.yaml job.yaml -i 2.yaml var priority: 1.yaml < job.yaml < 2.yaml case 3: submit -i 1.yaml job1.yaml -i 2.yaml job2.yaml -i 3.yaml var priority: 1.yaml < job1.yaml < 2.yaml < 3.yaml var priority: 1.yaml < 2.yaml < job2.yaml < 3.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 +++++++++++++---- sbin/submit | 30 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..640b25a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,8 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :default_yamls + attr_accessor :override_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +224,8 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @default_yamls = [] + @override_yamls = [] end def source_file_symkey(file) @@ -254,8 +258,11 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + + revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? + revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? + revise_hash(hash, @overrides, true) unless @overrides.empty? + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +293,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls(include_yamls) + include_hash = {} + include_hash[comment_to_symbol("#{include_yamls.join(' ')}")] = nil + include_hash.merge!(load_yaml_merge(include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..e8b17648 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -66,6 +71,7 @@ options = OptionParser.new do |opts| end +ARGV_CLONE = ARGV.clone options.parse!(ARGV) seen_job_yaml = false @@ -91,6 +97,19 @@ end job_ids = [] job_hash_list = [] +def prioritize_include_yaml(include_yamls, jobfile) + default_yamls, override_yamls = [], [] + jobfile_index = ARGV_CLONE.index(jobfile) + include_yamls.each do |yaml| + if ARGV_CLONE.index(yaml) < jobfile_index + default_yamls << yaml + else + override_yamls << yaml + end + end + return default_yamls, override_yamls +end + def find_jobfile(jobfile) return jobfile if File.exist?(jobfile) @@ -103,14 +122,25 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" ARGV.each do |jobfile| + default_yamls, override_yamls = prioritize_include_yaml(opt_include_yamls, jobfile) jobfile = find_jobfile(jobfile) jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.default_yamls = find_jobfiles(default_yamls) + jobs.override_yamls = find_jobfiles(override_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

1 0

[PATCH compass-ci] container/add-software: if cd failed, don't go on exec rm cmd
by Xu Xijian 21 Jan '21

21 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index aab659b..8a1d9ea 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -42,7 +42,7 @@ install_cmds=( addsoftware:auto bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages - cd /mnt/root + cd /mnt/root && rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} > /dev/null 2>&1 " ) -- 2.23.0

2 2

[PATCH compass-ci] delimiter: use check_job_credible API to replace credible? function
by Cao Xueliang 21 Jan '21

21 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 2c6549c..41b65de 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -78,6 +78,7 @@ module Utils # monitor the job id and job state query job stats when job state is extract_finished # according to the job stats return good/bad/nil def get_job_status(job, error_id) + bad_job_id = job['bad_job_id'] new_job_id = submit_job(job) puts "new job id: #{new_job_id}" return nil unless new_job_id @@ -86,7 +87,9 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? - raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + check_result = AssistResult.new.check_job_credible(bad_job_id, new_job_id, error_id) + raise "check job credible failed: #{bad_job_id}, #{new_job_id}, #{error_id}" if check_result == nil + raise "the job is incredible for bisect: #{new_job_id}" unless check_result['credible'] stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/5] new reset-tbox job yaml and script
by Zhang Dewan 21 Jan '21

21 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 2 +- tests/reset-tbox | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 216a3852..52ea76b7 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -7,4 +7,4 @@ docker_image: reset-tbox:latest queue: reset-tbox reset-tbox: - host_name: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox index 36aa37d5..4d2c191e 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -1,5 +1,5 @@ #!/bin/bash -# - host_nameset +# host_name account_file=/etc/lab/.account.info hosts_file=/etc/lab/hosts_file -- 2.23.0

2 2

[PATCH lkp-tests] lkp-bootstrap.service: fix killmode error
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] Welcome to openEuler 20.09! /usr/lib/systemd/system/lkp-bootstrap.service:11: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the ser vice. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. [How] config the lkp-bootstrap.service: KillMode=control-group Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service index 842c4aef8..a9eb5152a 100644 --- a/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service +++ b/rootfs/addon/usr/lib/systemd/system/lkp-bootstrap.service @@ -8,7 +8,7 @@ Type=forking Environment=HOME=/root ExecStart=/etc/init.d/lkp-bootstrap IgnoreSIGPIPE=no -KillMode=none +KillMode=control-group TasksMax=infinity [Install] -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: optimize borrow-machine.zh.md
by Luan Shengde 21 Jan '21

21 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/borrow-machine.zh.md | 304 ++++++++------------------------ 1 file changed, 75 insertions(+), 229 deletions(-) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 916aea7..bf85f97 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -1,243 +1,89 @@ -# 前提条件 - -请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… -- send apply account email. -- receive email from compass-ci-robot(a)qq.com. -- local environment configuration. - - -# 申请测试机（虚拟机） - -1. 生成本地RSA公私钥对 - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. 根据需求选择yaml - - 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. 提交yaml并连接测试机（虚拟机） - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170593 - query=>{"job_id":["z9.170593"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} - {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} - {"job_id": "z9.170593", "job_state": "boot"} - {"job_id": "z9.170593", "job_state": "download"} - {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - {"job_state":"running","job_id":"z9.170593"} - {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 - - - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 - - System information as of time: Thu Nov 26 06:44:18 CST 2020 - - System load: 0.83 - Processes: 107 - Memory used: 6.1% - Swap used: 0.0% - Usage On: 89% - IP address: 172.18.156.13 - Users online: 1 - - - - root@vm-2p8g ~# - ``` - - 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 - -4. 使用完毕退还测试机（虚拟机） - - ```shell - root@vm-2p8g ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` - - -# 申请测试机（物理机） - - -1. 生成本地RSA公私钥对 - - ```shell - hi684@account-vm ~% ssh-keygen -t rsa - Generating public/private rsa key pair. - Enter file in which to save the key (/home/hi684/.ssh/id_rsa): - Created directory '/home/hi684/.ssh'. - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /home/hi684/.ssh/id_rsa. - Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. - The key fingerprint is: - SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm - The key's randomart image is: - +---[RSA 2048]----+ - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - |xxxxxxxxxxxxxxxxx| - +----[SHA256]-----+ - hi684@account-vm ~% ls -hla .ssh - total 16K - drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . - drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. - -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa - -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub - ``` - -2. 根据需求选择yaml - - 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 - - ```shell - hi684@account-vm ~% cd lkp-tests/jobs - hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* - -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml - -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml - -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml - ``` - -3. 提交yaml并连接测试机（物理机） - - ```shell - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml - submit borrow-1h.yaml, got job_id=z9.170594 - query=>{"job_id":["z9.170594"]} - connect to ws://172.168.131.2:11310/filter - {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} - {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} - {"job_id": "z9.170594", "job_state": "boot"} - {"job_id": "z9.170594", "job_state": "download"} - {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} - {"job_state":"running","job_id":"z9.170594"} - {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} - Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts - Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. - Last login: Wed Sep 23 11:10:58 2020 +# 前置准备 +- 申请account +- 配置默认yaml文件 - Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 +如果未完成以上步骤，请参考[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)完成操作。 - System information as of time: Thu Nov 26 14:51:59 CST 2020 +# 申请测试机 - System load: 1.31 - Processes: 1020 - Memory used: 5.1% - Swap used: 0.0% - Usage On: 3% - IP address: 172.168.178.48 - Users online: 1 - - - - root@taishan200-2280-2s64p-256g--a5 ~# - ``` - - 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 - -4. 使用完毕退还测试机（物理机） - - ```shell - root@taishan200-2280-2s64p-256g--a5 ~# reboot - Connection to 172.168.131.2 closed by remote host. - Connection to 172.168.131.2 closed. - hi684@account-vm ~/lkp-tests/jobs% - ``` +## 1. 生成本地公/私钥对 + 使用下面命令查看是否已存在公/私钥： + + ''' + ls -a ~/.ssh + ''' + + 如果当前没有现成的公/私钥对，请使用下面命令进行生成： + + ''' + ssh-keygen -r rsa + ''' + +## 2. 选择job yaml + + 在 ~/lkp-tests/jobs 目录下未您提供了众多job yaml模板。 + + 使用以下命令可筛选出借用机器的模板： + + ''' + cd ~/lkp-tests/jobs + ls -a borrow* + ''' + + - 检查并确保您选择的模板中引用的公钥地址和文件名和你本地的公钥一致。 + +## 3. 提交job + + 使用以下命令提交job： + + ''' + submit -c -m testbox=vm-2p8g borrow-1h.yaml + ''' + + - 使用该命令，您可以实时查看job状态，测试机正常运行后将直接登入。 + +## 4. 退还测试机 + + 到期自动归还： + + 借用期限到达后，测试接将自动重启归还。 + + 手动归还： + + 执行‘reboot’命令重启测试机，测试机将立即归还。 + + - 所有测试机在执行‘reboot’命令后都会被归还，归还后不可再次登录使用。 + - 机器归还后，如果您还需要继续使用测试机，请重新提交job申请新的测试机。 # FAQ +## 自定义借用时长 + + 在借用机器的yaml模板中，找到’runtime‘字段，根据需求修改借用时长。 + +## submit命令指导 + + 学习submit命令，您可以使用以下命令查看submit命令的各项参数及使用方法： + + ''' + submit -h + ''' + + 也可以参考submit命令手册学习submit命令高级用法： + + [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + +## 可选的testbox -* 如何自行修改申请时长 - - ```shell - hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml - suite: borrow - testcase: borrow - - runtime: 1h - ssh_pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: - hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml - sleep: 1h - # 使用vim来修改你的runtime字段的值 - hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml - # 修改完毕后重新submit即可 - hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml - ``` - -* Submit命令指导 - - 参考文档：[submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - -* testbox有什么可选项 - - testbox可选项请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + 可选的testbox模板，请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts >![](./../public_sys-resources/icon-note.gif) **说明：** > - > 虚拟机的testbox : vm-xxx - > 物理机的testbox : taishan200-2280-xxx + > - 容器: dc-xxx + > - 虚拟机: vm-xxx + > - 物理机: taishan200-2280-xxx @@ -245,6 +91,6 @@ > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 -* 如何 borrow 指定的操作系统 +## 如何 borrow 指定的操作系统关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) -- 2.23.0

2 2

[PATCH lkp-tests] iso2rootfs: add step: archive checksum file to every rootfs if needed
by Yu Chuan 21 Jan '21

21 Jan '21

[Why] dailybuild have many version, we only store the checksum file of latest version, it's not enough, and it's not easy to identify every version. [How] So after the rootfs is generated, copy the checksum file corresponding to each rootfs to rootfs:/. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index af164bd43077..cb077fa28b0f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -576,6 +576,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: fix to download *.tar packages failure
by Liu Shaofei 21 Jan '21

21 Jan '21

[why] The server does not support HTTP protocol address download, only use the original HTTPS protocol address download. curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104 ==> ERROR: Failure while downloading buddy-2.4.tar.gz Aborting... Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..7c65f26bc 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -127,7 +127,6 @@ replace_source() } } } - url=$(echo "$url" | sed 's|https://|http://|g') echo \'$url\' >> $1 done -- 2.23.0

2 2

[PATCH v3 lkp-tests] sbin/submit: add parameter option -i for submit
by Wei Jihui 21 Jan '21

21 Jan '21

submit -i is for add other yamls content to job.yaml [usage] case 1: submit -i 1.yaml -i 2.yaml job.yaml var priority: job.yaml > 2.yaml > 1.yaml case 2: submit -i 1.yaml job.yaml -i 2.yaml var priority: 2.yaml > job.yaml > 1.yaml case 3: submit -i 1.yaml job1.yaml -i 2.yaml job2.yaml -i 3.yaml var priority: 3.yaml > job2.yaml > 2.yaml > 1.yaml var priority: 3.yaml > 2.yaml > job1.yaml > 1.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 +++++++++++++---- sbin/submit | 30 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..640b25a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,8 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :default_yamls + attr_accessor :override_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +224,8 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @default_yamls = [] + @override_yamls = [] end def source_file_symkey(file) @@ -254,8 +258,11 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + + revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? + revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? + revise_hash(hash, @overrides, true) unless @overrides.empty? + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +293,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls(include_yamls) + include_hash = {} + include_hash[comment_to_symbol("#{include_yamls.join(' ')}")] = nil + include_hash.merge!(load_yaml_merge(include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..e8b17648 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -66,6 +71,7 @@ options = OptionParser.new do |opts| end +ARGV_CLONE = ARGV.clone options.parse!(ARGV) seen_job_yaml = false @@ -91,6 +97,19 @@ end job_ids = [] job_hash_list = [] +def prioritize_include_yaml(include_yamls, jobfile) + default_yamls, override_yamls = [], [] + jobfile_index = ARGV_CLONE.index(jobfile) + include_yamls.each do |yaml| + if ARGV_CLONE.index(yaml) < jobfile_index + default_yamls << yaml + else + override_yamls << yaml + end + end + return default_yamls, override_yamls +end + def find_jobfile(jobfile) return jobfile if File.exist?(jobfile) @@ -103,14 +122,25 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" ARGV.each do |jobfile| + default_yamls, override_yamls = prioritize_include_yaml(opt_include_yamls, jobfile) jobfile = find_jobfile(jobfile) jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.default_yamls = find_jobfiles(default_yamls) + jobs.override_yamls = find_jobfiles(override_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

3 2

[PATCH v12 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 21 Jan '21

21 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 24 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.last_state.test.iperf.exit_code.127: 12 sum.stats.stderr./lkp/lkp/src/lib/env.sh:#:set_perf_path: 12 sum.stats.last_state.is_incomplete_run: 12 sum.stats.kmsg.timestamp:last: 45 sum.stats.last_state.exit_fail: 20 sum.stats.stderr.has_stderr: 44 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 202 +++++++++++++++++++++++++++---------------------- 1 file changed, 112 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..6051e22 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,159 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) - - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value + def set_job_summary(stats, job) + summary_result = '' + stats.keys.each do |stat| + if stat.match(/\.warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + if stat.match(/stderr\./i) + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + if stat.match(/error|\.exit_code/i) + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stat.match(/\.fail$/i) + job['summary.any_fail'] = 1 + summary_result = 'fail' end - - next unless job['stats'] - - @result[field] << job['stats'][field] if job['stats'][field] end - end - - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + return unless summary_result.empty? - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) + job['summary.success'] = 1 + end - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + set_job_summary(stats, job) end - - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + metrics.concat(stats.keys) end - return level + metrics.uniq! end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end - key = job_id + '.' + key - @stats_filter_result[key] = value + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - break + stats.each do |key, value| + result['raw.stats'][key] << value end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + end + end - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 1

[PATCH v2 compass-ci 1/5] upload-libvirt-xml/build: add upload-libvirt-xml docker image
by cuiyili 21 Jan '21

21 Jan '21

Add upload-libvirt-xml docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/build | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 container/upload-libvirt-xml/build diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build new file mode 100755 index 0000000..0490e15 --- /dev/null +++ b/container/upload-libvirt-xml/build @@ -0,0 +1,7 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t upload-libvirt-xml . -- 2.23.0

2 2

[PATCH v1 compass-ci 3/4] providers/libvirt: request scheduler to consume job
by Xiao Shenwei 21 Jan '21

21 Jan '21

provider a public methods: request_job it will register hostname mac queues and request http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 81 +++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..6cfa448 --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,81 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require_relative 'base' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer < Base + attr_reader :info + + def initialize(hostname, queues) + @info = {} + @info['hostname'] = hostname + @info['queues'] = queues + end + + def close + del_host_info + end + + def request_job + get_mac_from_hostname(@info['hostname']) + config_scheduler + set_host_info + host_exists + url = "http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}" + @logger.info("Request URL: #{url}") + res = %x(curl #{url}) + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + private + + def config_scheduler + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def get_mac_from_hostname(hostname) + cmd = %(echo #{hostname} | md5sum | sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/") + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def set_host_info + system "curl -X PUT \ + 'http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}'" + system "curl -X PUT \ + 'http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}'" + end + + def del_host_info + system "curl -X PUT 'http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}'" + system "curl -X PUT 'http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}'" + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci 5/5] upload-libvirt-xml/nginx.conf: nginx.conf in upload-libvirt-xml container
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/nginx.conf | 43 +++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt-xml/nginx.conf diff --git a/container/upload-libvirt-xml/nginx.conf b/container/upload-libvirt-xml/nginx.conf new file mode 100644 index 0000000..aef8fb8 --- /dev/null +++ b/container/upload-libvirt-xml/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt-xml"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

1 0

[PATCH v2 compass-ci 4/5] upload-libvirt-xml/openresty.sh: start openresty in upload-libvirt-xml docker image
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/openresty.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/upload-libvirt-xml/openresty.sh diff --git a/container/upload-libvirt-xml/openresty.sh b/container/upload-libvirt-xml/openresty.sh new file mode 100755 index 0000000..6215d91 --- /dev/null +++ b/container/upload-libvirt-xml/openresty.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +umask 002 +/usr/local/openresty/bin/openresty -g 'daemon off;' -- 2.23.0

1 0

[PATCH v2 compass-ci 3/5] upload-libvirt-xml/start: start upload-libvirt-xml docker container
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/upload-libvirt-xml/start diff --git a/container/upload-libvirt-xml/start b/container/upload-libvirt-xml/start new file mode 100755 index 0000000..ddb04e7 --- /dev/null +++ b/container/upload-libvirt-xml/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +: ${UPLOAD_LIBVIRT_PORT:=3081} + +docker_rm upload-libvirt-xml + +cmd=( + docker run -d + --name upload-libvirt-xml + --restart=always + -p ${UPLOAD_LIBVIRT_PORT}:3081 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml + upload-libvirt-xml +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/5] upload-libvirt-xml/Dockerfile: add upload-libvirt-xml docker image
by cuiyili 21 Jan '21

21 Jan '21

Add upload-libvirt-xml docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt-xml/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/upload-libvirt-xml/Dockerfile diff --git a/container/upload-libvirt-xml/Dockerfile b/container/upload-libvirt-xml/Dockerfile new file mode 100644 index 0000000..db48bcb --- /dev/null +++ b/container/upload-libvirt-xml/Dockerfile @@ -0,0 +1,14 @@ +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM openresty/openresty:alpine + +RUN apk add bash +RUN mkdir -p /var/log/nginx/ +RUN adduser lkp -u 1090 -g 1090 -D + +COPY ./openresty.sh /usr/local/bin/ + +CMD ["/usr/local/bin/openresty.sh"] -- 2.23.0

1 0

[PATCH lkp-tests] submit: upload libvirt config files
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt File's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job2sh.rb | 10 ++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7f9cf562f..14c4e3f4b 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -15,6 +15,16 @@ SHELL_BLOCK_KEYWORDS = { 'function' => ['{', '}'] }.freeze +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def valid_shell_variable?(key) key =~ /^[a-zA-Z_]+[a-zA-Z0-9_]*$/ end diff --git a/sbin/submit b/sbin/submit index 39cdcf576..e9e10d903 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

2 3

[PATCH v2 compass-ci] lib/job.cr: add log when check account failed
by Wu Zhende 21 Jan '21

21 Jan '21

[Why] The information about the job that fails the account verification can be traced on kibana. [Log] {"level_num":2,"level":"WARN","time":"2021-01-21T11:28:25.476+0800", "msg":"Invalid account","my_email":"wuzhende666(a)163.com","my_name":"wuzhende","suite":"iperf","testbox":vm-2p8g"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 17a0f1b..9b52d88 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -10,6 +10,7 @@ require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" require "../scheduler/elasticsearch_client" +require "./json_logger" struct JSON::Any def []=(key : String, value : String) @@ -53,6 +54,7 @@ class Job @hash = job_content.as_h @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new + @log = JSONLogger.new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" @@ -394,17 +396,30 @@ class Job error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise error_msg unless account_info.is_a?(JSON::Any) + + flag = is_valid_account?(account_info) + @log.warn({"msg" => "Invalid account", + "my_email" => self["my_email"], + "my_name" => self["my_name"], + "suite" => self["suite"], + "testbox" => self["testbox"] + }.to_json) unless flag + raise error_msg unless flag + + @hash.delete("my_uuid") + @hash.delete("my_token") + end + + private def is_valid_account?(account_info) + return false unless account_info.is_a?(JSON::Any) @account_info = account_info.as_h # my_name can be nil in es # my_token can't be nil in es - raise error_msg unless self["my_name"] == @account_info["my_name"]?.to_s - raise error_msg unless self["my_token"] == @account_info["my_token"]? - - @hash.delete("my_uuid") - @hash.delete("my_token") + return false unless self["my_name"] == @account_info["my_name"]?.to_s + return false unless self["my_token"] == @account_info["my_token"]? + return true end private def get_initialized_keys -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: add -f option for uploading libvirt config
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt FILE's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.rb | 11 +++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc98..812f8c299 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -16,6 +16,17 @@ require 'set' require 'pp' require 'English' +# copied from compass-ci/container/defconfig.rb +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def restore(ah, copy) if ah.class == Hash ah.clear.merge!(copy) diff --git a/sbin/submit b/sbin/submit index 39cdcf576..e9e10d903 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

1 1

[PATCH compass-ci] lib/job.cr: add log when check account failed
by Wu Zhende 21 Jan '21

21 Jan '21

[Why] The information about the job that fails the account verification can be traced on kibana. [Log] {"level_num":2,"level":"WARN","time":"2021-01-21T11:08:19.647+0800","msg":"Invalid account","my_email":"wuzhende666(a)163.com","suite":"iperf","testcase":"iperf"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 17a0f1b..2971a77 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -10,6 +10,7 @@ require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" require "../scheduler/elasticsearch_client" +require "./json_logger" struct JSON::Any def []=(key : String, value : String) @@ -53,6 +54,7 @@ class Job @hash = job_content.as_h @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new + @log = JSONLogger.new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" @@ -394,17 +396,29 @@ class Job error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise error_msg unless account_info.is_a?(JSON::Any) + + flag = is_valid_account?(account_info) + @log.warn({"msg" => "Invalid account", + "my_email" => self["my_email"], + "suite" => self["suite"], + "testcase" => self["testcase"] + }.to_json) unless flag + raise error_msg unless flag + + @hash.delete("my_uuid") + @hash.delete("my_token") + end + + private def is_valid_account?(account_info) + return false unless account_info.is_a?(JSON::Any) @account_info = account_info.as_h # my_name can be nil in es # my_token can't be nil in es - raise error_msg unless self["my_name"] == @account_info["my_name"]?.to_s - raise error_msg unless self["my_token"] == @account_info["my_token"]? - - @hash.delete("my_uuid") - @hash.delete("my_token") + return false unless self["my_name"] == @account_info["my_name"]?.to_s + return false unless self["my_token"] == @account_info["my_token"]? + return true end private def get_initialized_keys -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: add new_refs_count by year/month/day
by Li Yuanchao 21 Jan '21

21 Jan '21

Add a hash table new_refs_count in fork_stat for every repo, which will record new refs times of this repo by year/month/day. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 4f74751..6d1624f 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -298,6 +298,7 @@ class MirrorMain offset_new_refs = 0 if offset_new_refs >= 10 @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) end def update_fork_stat(git_repo, possible_new_refs) @@ -373,15 +374,53 @@ class MirrorMain fetch_time: [], offset_fetch: 0, new_refs_time: [], - offset_new_refs: 0 + offset_new_refs: 0, + new_refs_count: {} } query = { query: { match: { git_repo: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? fork_stat.each_key do |key| - fork_stat[key] = result['hits'][0]['_source'][key.to_s] + fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end return fork_stat end + + def create_year_hash(new_refs_count, year, month, day) + new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + return new_refs_count + end + + def update_year_hash(new_refs_count, year, month, day) + new_refs_count[year]['year_count'] += 1 + return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + + return update_month_hash(new_refs_count, year, month, day) + end + + def create_month_hash(new_refs_count, year, month, day) + new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + return new_refs_count + end + + def update_month_hash(new_refs_count, year, month, day) + new_refs_count[year][month]['month_count'] += 1 + if new_refs_count[year][month][day].nil? + new_refs_count[year][month][day] = 1 + else + new_refs_count[year][month][day] += 1 + end + return new_refs_count + end + + def update_new_refs_count(new_refs_count) + t = Time.now + year = t.year.to_s + month = t.month.to_s + day = t.day.to_s + return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? + + return update_year_hash(new_refs_count, year, month, day) + end end -- 2.23.0

1 0

[PATCH compass-ci] container/assist-result: add check_job_credible function
by Cao Xueliang 21 Jan '21

21 Jan '21

The container assist-result mount "/srv/result" dir, the function credible? need "/srv/result" dir, so it can be implemented in the container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assist-result/routes.rb | 12 ++++++++++++ .../assist-result/views/check_job_credible.rb | 9 +++++++++ lib/assist_result_client.rb | 15 +++++++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 container/assist-result/views/check_job_credible.rb diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 47ef92f..60a2fbf 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -12,4 +12,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport json yaml elasticsearch sinatra puma + gem install rest-client activesupport json yaml elasticsearch sinatra puma terminal-table diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 1013643..077a427 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/check_job_credible' configure do set :bind, '0.0.0.0' @@ -22,3 +23,14 @@ get '/get_job_yaml/:job_id' do return [200, result.to_json] end + +post '/check_job_credible' do + begin + data = JSON.parse(Base64.decode64(request.body.read)) + result = check_job_credible(data['pre_job_id'], data['cur_job_id'], data['error_id']) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, {'credible' => result}.to_json] +end diff --git a/container/assist-result/views/check_job_credible.rb b/container/assist-result/views/check_job_credible.rb new file mode 100644 index 0000000..a8d387f --- /dev/null +++ b/container/assist-result/views/check_job_credible.rb @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/compare_error_messages" + +def check_job_credible(pre_job_id, cur_job_id, error_id) + return credible?(pre_job_id, cur_job_id, error_id) +end diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb index 042b189..fcf5d9d 100644 --- a/lib/assist_result_client.rb +++ b/lib/assist_result_client.rb @@ -3,6 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'json' +require 'base64' require 'rest-client' require_relative 'constants' @@ -19,4 +21,17 @@ class AssistResult return response.body end + + def check_job_credible(pre_job_id, cur_job_id, error_id) + data = { + 'pre_job_id' => pre_job_id, + 'cur_job_id' => cur_job_id, + 'error_id' => error_id + } + resource = RestClient::Resource.new("http://#{@host}:#{@port}/check_job_credible") + response = resource.post(Base64.encode64(data.to_json)) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH compass-ci] /container/dracut-initrd/bin: refactor the code
by Xu Xijian 21 Jan '21

21 Jan '21

1. delete the redundant space before "-d" of "[ -d xx ]" 2. add blank line after "continue" cmd 3. turn some 2 line code block into one line to be more compact Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/overlay-lkp.sh | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/container/dracut-initrd/bin/overlay-lkp.sh b/container/dracut-initrd/bin/overlay-lkp.sh index 1f5819b..9a2800e 100644 --- a/container/dracut-initrd/bin/overlay-lkp.sh +++ b/container/dracut-initrd/bin/overlay-lkp.sh @@ -2,7 +2,7 @@ # SPDX-License-Identifier: GPL-2.0 # transfer LKP dirs -[ -d /lkp ] || return 0 +[ -d /lkp ] || return 0 cp -a /lkp "$NEWROOT"/ @@ -11,8 +11,8 @@ do dir=$(basename "$i") [ "$i" != "${i%/\*}" ] && continue # skip: i='dir/*' - [ -d "$NEWROOT/$dir" ] || - mkdir -p "$NEWROOT/$dir" + + [ -d "$NEWROOT/$dir" ] || mkdir -p "$NEWROOT/$dir" for j in "$i"/* "$i"/.??* do @@ -25,8 +25,7 @@ do subdir=$(basename "$j") - [ -d "$NEWROOT/$dir/$subdir" ] || - mkdir -p "$NEWROOT/$dir/$subdir" + [ -d "$NEWROOT/$dir/$subdir" ] || mkdir -p "$NEWROOT/$dir/$subdir" for k in "$j"/* do @@ -37,7 +36,7 @@ do done done -[ -d /usr/src ] && +[ -d /usr/src ] && cp -a /usr/src "$NEWROOT"/usr/ kmdir=/lib/modules/$(uname -r) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 21 Jan '21

21 Jan '21

[why] nokogiri dependent zlib-devel Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..7f3d982 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/nginx.conf: nginx.conf in upload-libvirt container
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/nginx.conf | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt/nginx.conf diff --git a/container/upload-libvirt/nginx.conf b/container/upload-libvirt/nginx.conf new file mode 100644 index 0000000..6dea3d3 --- /dev/null +++ b/container/upload-libvirt/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

2 1

[PATCH v4 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..411c130 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0

[PATCH v3 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..e161b17 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 1

[PATCH v2 lkp-tests] pkg: add PKGBUILD for openeuler docker test
by Bai Jing 20 Jan '21

20 Jan '21

[why] The source file is missing in the openEuler image, So we need to add source files for it. Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/openeuler_docker/PKGBUILD | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 pkg/openeuler_docker/PKGBUILD diff --git a/pkg/openeuler_docker/PKGBUILD b/pkg/openeuler_docker/PKGBUILD new file mode 100644 index 000000000..6a1190718 --- /dev/null +++ b/pkg/openeuler_docker/PKGBUILD @@ -0,0 +1,27 @@ +pkgname=openeuler_docker +pkgver=1.0 +pkgrel=1 +pkgdesc="openeuler docker image" +arch=('i386' 'x86_64') +url="https://repo.openeuler.org/openEuler-20.03-LTS/docker_img" +license=('GPL') +source=("https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler…") +md5sums=('6aa300910c9f9040b725939fb80e8121') + +handle_docker_image() +{ + docker load < openEuler-docker.aarch64.tar.xz + docker run -d --name my-openeuler openeuler-20.03-lts:latest + container_id=$(docker ps -aqf 'name=my-openeuler') + docker cp /etc/yum.repos.d/openEuler.repo $container_id:/etc/yum.repos.d/ + docker commit $container_id openeuler-20.03-lts:v1 + docker save -o openeuler-20.03-lts-v1.tar openeuler-20.03-lts:v1 + docker rm -f $container_id +} + +package() +{ + handle_docker_image + mkdir -p "$pkgdir/lkp/benchmarks/${pkgname}" + cp "$srcdir/openeuler-20.03-lts-v1.tar" "$pkgdir/lkp/benchmarks/${pkgname}" +} -- 2.23.0

1 0

[PATCH v4 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 20 Jan '21

20 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..4f74751 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,23 @@ class MirrorMain handle_submodule(submodule) end + + def get_fork_stat(git_repo) + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return fork_stat unless result['total'].positive? + + fork_stat.each_key do |key| + fork_stat[key] = result['hits'][0]['_source'][key.to_s] + end + return fork_stat + end end -- 2.23.0

1 0

[PATCH v3 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 20 Jan '21

20 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..6bb474a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,28 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + es_data.each do |key, value| + fork_stat[key.to_sym] = value + end + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 20 Jan '21

20 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..93977ec15438 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,6 +6,9 @@ : ${nr_cpu:=1} : ${memory:=1G} +: ${LKP_SRC:="/c/lkp-tests"} + +source ${LKP_SRC}/lib/log.sh check_logfile() { @@ -52,15 +55,23 @@ parse_ipxe_script() check_option_value() { + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 + } + [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + log_error "Can not find kernel file or kernel file is empty: $kernel." + exit 1 } # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || { + log_error "Can not find any initrd for current vm." + exit 1 + } } set_initrd() -- 2.23.0

3 3

[PATCH v2 compass-ci 2/2] service/scheduler: save testbox info in es
by Wu Zhende 20 Jan '21

20 Jan '21

When a testbox get a job, record its information in es. The deadline field indicates the time when the job executed by the testbox times out. It's a sign that other measures are needed to intervene. Can use curl -XGET http://localhost:9200/testbox/_doc/_search -d '{"query":{"range":{"deadline":{"lt":"now"}}}}' to find all timeout testboxs and jobs Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 19 +++++++++++++++++++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 20 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51e0d74..8d7c218 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -134,6 +134,25 @@ class Sched @log.warn(e) end + def set_tbox_boot_wtmp(job : Job) + time = Time.local + booting_time = time.to_s("%Y-%m-%dT%H:%M:%S") + + runtime = (job["timeout"]? || job["runtime"]?).to_s + runtime = 1800 if runtime.empty? + + # reserve 300 seconds for system startup, hw machine will need such long time + deadline = (time + (runtime.to_i32 * 2 + 300).second).to_s("%Y-%m-%dT%H:%M:%S") + hash = { + "job_id" => job["id"], + "state" => "booting", + "booting_time" => booting_time, + "deadline" => deadline + } + + @es.update_tbox(job["testbox"], hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b7f3d75..f8d7caa 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -101,6 +101,7 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + set_tbox_boot_wtmp(job) else # for physical machines spawn { auto_submit_idle_job(host) } -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] scheduler/elasticsearch_client.cr: support update all hash
by Wu Zhende 20 Jan '21

20 Jan '21

[Before] Only support update job_id/job_state/history [After] Support update any key-value. Because need save other new key-value in testbox. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index b17d7f8..c1663c9 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -3,6 +3,7 @@ require "yaml" require "json" +require "any_merge" require "elasticsearch-crystal/elasticsearch/api" require "./constants" require "../lib/job" @@ -89,21 +90,30 @@ class Elasticsearch::Client def update_tbox(testbox : String, wtmp_hash : Hash) query = {:index => "testbox", :type => "_doc", :id => testbox} - result = @client.get_source(query) if @client.exists(query) - history = result["history"].as_a? if result.is_a?(JSON::Any) + if @client.exists(query) + result = @client.get_source(query) + raise result unless result.is_a?(JSON::Any) + + history = result["history"].as_a? + result = result.as_h + else + result = wtmp_hash + end + history ||= [] of JSON::Any history << JSON.parse(wtmp_hash.to_json) + history = JSON.parse(history.to_json) + + body = { "history" => history} + body.any_merge!(result) + body.any_merge!(wtmp_hash) @client.create( { :index => "testbox", :type => "_doc", :id => testbox, - :body => { - :state => wtmp_hash["state"], - :job_id => wtmp_hash["job_id"], - :history => history - } + :body => body } ) end -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: fix script hangs when no job now
by Liu Yinsi 20 Jan '21

20 Jan '21

[error] when no job now, $initrds is null, execute my-qemu.sh will hangs. lys@crystal ~/compass-ci/providers% ./my-qemu.sh hostname: vm-2p8g.lys-3917841 mac: 0a-4b-33-a0-96-69 DoneDone % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 79 100 79 0 0 19750 0 --:--:-- --:--:-- --:--:-- 19750 lys@crystal ~/compass-ci/providers/qemu% cat --help cat Copy standard input to standard output. when input is null, script will hang to waiting for input. [how] function set_options(): cat $initrds > $initrd before: execut cat command, then check after: check kernel, check initrds, if not null, execute cat command. in addition, check_option_value -> check_kernel, check_initrd, check_qemu convenient to call as needed. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 4d05964..b4e87fc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() done < $ipxe_script } -check_option_value() +check_kernel() { [ -n "$kernel" ] || { log_info "Can not find job for current hostname: $hostname." @@ -64,10 +64,16 @@ check_option_value() log_error "Can not find kernel file or kernel file is empty: $kernel." exit 1 } - +} + +check_qemu() +{ # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} +check_initrds() +{ [ -n "$initrds" ] || { log_error "The current initrds is null." exit 1 @@ -118,6 +124,8 @@ set_qemu() do command -v "$qemu" > /dev/null && break done + + check_qemu } print_message() @@ -209,8 +217,10 @@ write_logfile parse_ipxe_script +check_kernel +check_initrds + set_options -check_option_value print_message -- 2.23.0

1 1

[PATCH v11 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 20 Jan '21

20 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 24 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.last_state.test.iperf.exit_code.127: 12 sum.stats.stderr./lkp/lkp/src/lib/env.sh:#:set_perf_path: 12 sum.stats.last_state.is_incomplete_run: 12 sum.stats.kmsg.timestamp:last: 45 sum.stats.last_state.exit_fail: 20 sum.stats.stderr.has_stderr: 44 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 202 +++++++++++++++++++++++++++---------------------- 1 file changed, 112 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..94c3193 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,159 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) - - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value + def set_job_summary(stats, job) + summary_result = '' + stats.keys.each do |stat| + if stat.match(/\.warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + if stat.match(/stderr\./i) + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + if stat.match(/error/i) + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stat.match(/\.fail$/i) + job['summary.any_fail'] = 1 + summary_result = 'fail' end - - next unless job['stats'] - - @result[field] << job['stats'][field] if job['stats'][field] end - end - - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + return unless summary_result.empty? - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) + job['summary.success'] = 1 + end - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + set_job_summary(stats, job) end - - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + metrics.concat(stats.keys) end - return level + metrics.uniq! end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end - key = job_id + '.' + key - @stats_filter_result[key] = value + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - break + stats.each do |key, value| + result['raw.stats'][key] << value end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + end + end - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] providers/qemu: fix script hangs when no job now
by Liu Yinsi 20 Jan '21

20 Jan '21

[error] when no job now, $initrds is null, execute my-qemu.sh will hangs. lys@crystal ~/compass-ci/providers% ./my-qemu.sh hostname: vm-2p8g.lys-3917841 mac: 0a-4b-33-a0-96-69 DoneDone % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 79 100 79 0 0 19750 0 --:--:-- --:--:-- --:--:-- 19750 lys@crystal ~/compass-ci/providers/qemu% cat --help cat Copy standard input to standard output. when input is null, script will hang to waiting for input. [how] before: execute cat command, then judge $initrds is null or not after: judge $initrds is null or not, if not null, execute cat command Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 4d05964..0c58d94 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -68,16 +68,17 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || { + if [ -n "$initrds" ]; then + cat $initrds > $initrd + else log_error "The current initrds is null." exit 1 - } + fi } set_initrd() { initrd=initrd - cat $initrds > $initrd } set_bios() -- 2.23.0

2 2

[PATCH v4 lkp-tests 5/5] new reset-tbox job yaml and script
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- tests/reset-tbox | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/reset-tbox b/tests/reset-tbox index 4d2c191e..37c23759 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -30,7 +30,7 @@ tbox_check() reset_tbox() { echo "reboot the tesetbox..." - ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset } result_check() @@ -38,7 +38,7 @@ result_check() if [ $? -ne 0 ] ; then echo "Failed to reboot the testbox: $host_name" fi - echo "done." + echo "done." } main() -- 2.23.0

1 0

[PATCH v4 lkp-tests 3/5] modify test/reset-tbox
by Zhang Dewan 20 Jan '21

20 Jan '21

at the test just now, "testbox: dc-8g" is need. Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- tests/reset-tbox | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/reset-tbox b/tests/reset-tbox index 1ca17bfd..36aa37d5 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -1,11 +1,11 @@ #!/bin/bash -# host_nameset +# - host_nameset account_file=/etc/lab/.account.info hosts_file=/etc/lab/hosts_file command_file=/usr/bin/ipmitool -FileCheck() +file_check() { for i in $account_file $hosts_file $command_file do @@ -17,7 +17,7 @@ FileCheck() source $account_file } -TboxCheck() +tbox_check() { ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') @@ -27,13 +27,13 @@ TboxCheck() } } -ResetTbox() +reset_tbox() { echo "reboot the tesetbox..." ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset } -ResultCheck() +result_check() { if [ $? -ne 0 ] ; then echo "Failed to reboot the testbox: $host_name" @@ -43,10 +43,10 @@ ResultCheck() main() { - FileCheck - TboxCheck - ResetTbox - ResultCheck + file_check + tbox_check + reset_tbox + result_check } main -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/5] alter reset-tbox.yaml
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 4d43850f..216a3852 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -3,7 +3,7 @@ category: functional testbox: dc-8g os_mount: container -docker_image: 172.168.131.2:5001/reset-tbox +docker_image: reset-tbox:latest queue: reset-tbox reset-tbox: -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/5] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

1 0

[PATCH compass-ci 5/5] upload-libvirt/nginx.conf: nginx.conf in upload-libvirt container
by cuiyili 20 Jan '21

20 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/nginx.conf | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt/nginx.conf diff --git a/container/upload-libvirt/nginx.conf b/container/upload-libvirt/nginx.conf new file mode 100644 index 0000000..6b2892b --- /dev/null +++ b/container/upload-libvirt/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

1 0

[PATCH compass-ci 4/5] upload-libvirt/start: start upload-libvirt docker container
by cuiyili 20 Jan '21

20 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/start | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 container/upload-libvirt/start diff --git a/container/upload-libvirt/start b/container/upload-libvirt/start new file mode 100755 index 0000000..5d149ad --- /dev/null +++ b/container/upload-libvirt/start @@ -0,0 +1,22 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +: ${UPLOAD_LIBVIRT_PORT:=3081} +docker_rm upload-libvirt + +cmd=( + docker run -d + --name upload-libvirt + --restart=always + -p ${UPLOAD_LIBVIRT_PORT}:3081 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml + upload-libvirt +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci 3/5] upload-libvirt/openresty.sh: start openresty in upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/openresty.sh | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/upload-libvirt/openresty.sh diff --git a/container/upload-libvirt/openresty.sh b/container/upload-libvirt/openresty.sh new file mode 100755 index 0000000..0a54b5e --- /dev/null +++ b/container/upload-libvirt/openresty.sh @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ + +umask 002 +/usr/local/openresty/bin/openresty -g 'daemon off;' -- 2.23.0

1 0

[PATCH compass-ci 2/5] upload-libvirt/build: add upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Add upload-libvirt docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/build | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 container/upload-libvirt/build diff --git a/container/upload-libvirt/build b/container/upload-libvirt/build new file mode 100755 index 0000000..bda6a3e --- /dev/null +++ b/container/upload-libvirt/build @@ -0,0 +1,7 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t upload-libvirt . -- 2.23.0

1 0

[PATCH compass-ci 1/5] upload-libvirt/Dockerfile: add upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Add upload-libvirt docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/upload-libvirt/Dockerfile diff --git a/container/upload-libvirt/Dockerfile b/container/upload-libvirt/Dockerfile new file mode 100644 index 0000000..db48bcb --- /dev/null +++ b/container/upload-libvirt/Dockerfile @@ -0,0 +1,14 @@ +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM openresty/openresty:alpine + +RUN apk add bash +RUN mkdir -p /var/log/nginx/ +RUN adduser lkp -u 1090 -g 1090 -D + +COPY ./openresty.sh /usr/local/bin/ + +CMD ["/usr/local/bin/openresty.sh"] -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/3] modify test/reset-tbox
by Zhang Dewan 20 Jan '21

20 Jan '21

at the test just now, "testbox: dc-8g" is need. Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- tests/reset-tbox | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/reset-tbox b/tests/reset-tbox index 1ca17bfd..36aa37d5 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -1,11 +1,11 @@ #!/bin/bash -# host_nameset +# - host_nameset account_file=/etc/lab/.account.info hosts_file=/etc/lab/hosts_file command_file=/usr/bin/ipmitool -FileCheck() +file_check() { for i in $account_file $hosts_file $command_file do @@ -17,7 +17,7 @@ FileCheck() source $account_file } -TboxCheck() +tbox_check() { ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') @@ -27,13 +27,13 @@ TboxCheck() } } -ResetTbox() +reset_tbox() { echo "reboot the tesetbox..." ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset } -ResultCheck() +result_check() { if [ $? -ne 0 ] ; then echo "Failed to reboot the testbox: $host_name" @@ -43,10 +43,10 @@ ResultCheck() main() { - FileCheck - TboxCheck - ResetTbox - ResultCheck + file_check + tbox_check + reset_tbox + result_check } main -- 2.23.0

2 2