- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] taskqueue: add build dependent container: redis
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] The container taskqueue depends on redis, so add redis to taskqueue/build-depends. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/taskqueue/build-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/build-depends b/container/taskqueue/build-depends index 5dd403e166af..a961b878fb44 100755 --- a/container/taskqueue/build-depends +++ b/container/taskqueue/build-depends @@ -1 +1,2 @@ scheduler-dev +redis -- 2.23.0

2 4

[PATCH v4 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 23 Jan '21

23 Jan '21

request url: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

2 1

[PATCH v4 compass-ci 2/3] libvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in this way, user can get message when error occurred [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin/quick_fetch: add an interface script for git_mirror speed up
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/quick_fetch | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 sbin/quick_fetch diff --git a/sbin/quick_fetch b/sbin/quick_fetch new file mode 100755 index 0000000..e8378ec --- /dev/null +++ b/sbin/quick_fetch @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +# usage: quick_fetch git_repo:$git_repo +# or: quick_fetch $url +# quick_fetch https://github.com/Siguyi/AvxToNeon +# quick_fetch git_repo:u/upstream-repos/upstream-repos +def send_url_of_repo + connection = Bunny.new('amqp://172.17.0.1:5672') + connection.start + channel = connection.create_channel + + queue = channel.queue('web_hook') + message = ARGV[0] + queue.publish(message) + connection.close +end + +send_url_of_repo -- 2.23.0

1 0

[PATCH compass-ci 1/2] git_mirror.rb: add one more parameter choice when use webhook
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6d1624f..a111e51 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -318,6 +318,8 @@ class MirrorMain # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) + return webhook_url.split(':')[1].gsub(' ', '') if webhook_url =~ /^(git_repo:)/ + fork_name, project = webhook_url.split('/')[-2, 2] git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] bvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in the fulture [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 in this way, user can perceived message when error occurred Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 2

[PATCH compass-ci] sparrow/README.md: improve readability.
by Li Ping 23 Jan '21

23 Jan '21

[why] Markdown does not translate every line break character in a paragraph into a <br/> tag. ref: http://markdown.cn/ obviously, in some paragraphs of this markdown document, the newline effect is not shown, content would be displayed as a single line, readability would be reduced. --- sparrow/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/README.md b/sparrow/README.md index c7450fd..1906f10 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -72,8 +72,10 @@ ``` 执行上述命令正常情况下会提示信息如下: + ``` submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 23 Jan '21

23 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 23 Jan '21

23 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: reformatting the document
by Li Ping 23 Jan '21

23 Jan '21

[why] In markdown documents, the newline effect is not shown, content not placed in ``` ''' would be displayed as a single line, readability would be reduced. Signed-off-by: Li Ping <1477412247(a)qq.com> --- sparrow/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/README.md b/sparrow/README.md index 794e433..3d7cdd0 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -72,8 +72,10 @@ ``` 执行上述命令正常情况下会提示信息如下: + ``` submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + ``` compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） -- 2.23.0

3 2

[PATCH v2 lkp-tests 3/3] distro/depends: add nginx-server-dev
by Wei Jihui 23 Jan '21

23 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server-dev | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/nginx-server-dev diff --git a/distro/depends/nginx-server-dev b/distro/depends/nginx-server-dev new file mode 100644 index 00000000..170af747 --- /dev/null +++ b/distro/depends/nginx-server-dev @@ -0,0 +1,2 @@ +pcre-devel +libpcre3-dev -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] pkg/nginx-server: add ngixn conf
by Wei Jihui 23 Jan '21

23 Jan '21

nginx-server test need forbidden nf_conntrack when os setup Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx-server/PKGBUILD | 10 ++++++++-- pkg/nginx-server/blacklist.conf | 10 ++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) create mode 100644 pkg/nginx-server/blacklist.conf diff --git a/pkg/nginx-server/PKGBUILD b/pkg/nginx-server/PKGBUILD index 141722e1..ac458776 100644 --- a/pkg/nginx-server/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -5,8 +5,8 @@ arch=('i386' 'x86_64') url="https://nginx.org" license=('BSD') nginx_name=("nginx-${pkgver}.${pkgrel}") -source=("https://nginx.org/download/${nginx_name}.tar.gz") -md5sums=('SKIP') +source=("https://nginx.org/download/${nginx_name}.tar.gz" "blacklist.conf") +md5sums=('SKIP' 'SKIP') build() { @@ -19,6 +19,12 @@ build() package() { + mkdir -p "$pkgdir/rootfs/addon/etc/modprobe.d/" + cp blacklist.conf "$pkgdir/rootfs/addon/etc/modprobe.d/" + + mkdir -p "$pkgdir/lkp/benchmarks/${pkgname}/" + cp -r "$LKP_SRC/pkg/${pkgname}/nginx_conf" "$pkgdir/lkp/benchmarks/${pkgname}/" + cd "${srcdir}/${nginx_name}" make install DESTDIR="${pkgdir}" } diff --git a/pkg/nginx-server/blacklist.conf b/pkg/nginx-server/blacklist.conf new file mode 100644 index 00000000..e8798cc7 --- /dev/null +++ b/pkg/nginx-server/blacklist.conf @@ -0,0 +1,10 @@ +install nf_conntrack /bin/false +blacklist nf_conntrack +blacklist nf_conntrack_ipv6 +blacklist xt_conntrack +blacklist nf_conntrack_ftp +blacklist xt_state +blacklist iptable_nat +blacklist ipt_REDIRECT +blacklist nf_nat +blacklist nf_conntrack_ipv4 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] pkg: rename nginx to nginx-server
by Wei Jihui 23 Jan '21

23 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/{nginx => nginx-server}/PKGBUILD | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) rename pkg/{nginx => nginx-server}/PKGBUILD (63%) diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx-server/PKGBUILD similarity index 63% rename from pkg/nginx/PKGBUILD rename to pkg/nginx-server/PKGBUILD index 989676c4..141722e1 100644 --- a/pkg/nginx/PKGBUILD +++ b/pkg/nginx-server/PKGBUILD @@ -1,23 +1,24 @@ -pkgname=nginx +pkgname=nginx-server pkgver=1.14 pkgrel=2 arch=('i386' 'x86_64') url="https://nginx.org" license=('BSD') -source=("https://nginx.org/download/nginx-${pkgver}.${pkgrel}.tar.gz") +nginx_name=("nginx-${pkgver}.${pkgrel}") +source=("https://nginx.org/download/${nginx_name}.tar.gz") md5sums=('SKIP') build() { benchmark_path="/lkp/benchmarks/${pkgname}" mkdir -p "${benchmark_path}" - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/${nginx_name}" ./configure --with-http_ssl_module --prefix="${benchmark_path}" make -j4 } package() { - cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + cd "${srcdir}/${nginx_name}" make install DESTDIR="${pkgdir}" } -- 2.23.0

1 0

[PATCH lkp-tests] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 22 Jan '21

22 Jan '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 1

[PATCH compass-ci] sparrow/1-storage: remove mount os rootfs
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] for locally deploy Compass CI with one click, we open directory /srv/os to download os rootfs, rather than open shared directory to mount. so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 3 --- 1 file changed, 3 deletions(-) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index eb7270d..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,3 @@ dirs=( ) mkdir -p "${dirs[@]}" - -mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr -mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/deploy-cci: mount os rootfs
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] mount our server's os rootfs directory is a dedicated way for $LKP_SRC/test/deploy-cci, it is not a general way for deploy Compass CI with one click, put it in tests/deploy-cci more reasonable, not in $CCI_SRC/sparrow script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index 8e6acbac4..e2a7bd320 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -35,5 +35,13 @@ deploy() ./install-tiny } +mount_rootfs() +{ + mkdir -p /srv/{os,initrd} + mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +} + export_env +mount_rootfs deploy -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/depends: fix failed to mount
by Liu Yinsi 22 Jan '21

22 Jan '21

[error] root@test-lys /c/compass-ci/container/os-cifs# mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr mount: /srv: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program. [why] missing depends cifs-utils causes mount error Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- distro/depends/deploy-cci | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/deploy-cci b/distro/depends/deploy-cci index 5664e303b..c0d6a5310 100644 --- a/distro/depends/deploy-cci +++ b/distro/depends/deploy-cci @@ -1 +1,2 @@ git +cifs-utils -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_error_messages.rb: return false when matching error filename failed
by Lin Jiaxin 22 Jan '21

22 Jan '21

If error_id = 'build-pkg.c++:fatal-error:switch'~x16~xe6~xab~xaa'does-not-start-with'-'', cannot get the error filename and return an empty hash. The empty hash is not credible too. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 2e35019..5f74be7 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -17,7 +17,9 @@ def credible?(previous_job_id, later_job_id, error_id) previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') - return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + filenames_check = filenames_check(previous_result_file, later_result_file, error_id) + + return false if filenames_check.empty? || filenames_check.value?(false) return true end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare_error_messages.rb: remove filename extension when matching error filename
by Lin Jiaxin 22 Jan '21

22 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 92533b8..2e35019 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -28,8 +28,8 @@ def filenames_check(previous_result_file, later_result_file, error_id) error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) error_lines.each do |error_line| - # "src/ssl_sock.c:1454:104: warning: unused parameter 'al' [-Wunused-parameter]" => "src/ssl_sock.c" - filenames << $1 if error_line =~ /(.*)(:\d+){2}: (error|warning):/ + # "src/ssl_sock.c:1454:104: warning: unused parameter 'al' [-Wunused-parameter]" => "src/ssl_sock" + filenames << $1.chomp(File.extname($1)) if error_line =~ /(.*)(:\d+){2}: (error|warning):/ end File.open(later_result_file).each_line do |line| -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 22 Jan '21

22 Jan '21

add limit for borrowing machine: runtime <= 10 days the job will break up if the value of runtime in the job yaml is greater than 10 days Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..bb9b792 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -64,6 +64,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + check_run_time() set_sshr_info() return end @@ -410,6 +411,15 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash["testcase"] == "borrow" + + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hash["runtime"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/dump-stats: support dump 'message'
by Lu Weitao 22 Jan '21

22 Jan '21

[Example] { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ], "iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/dump-stat | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/dump-stat b/sbin/dump-stat index b5abf5a9b..6ebda1f29 100755 --- a/sbin/dump-stat +++ b/sbin/dump-stat @@ -61,10 +61,8 @@ while (line = STDIN.gets) end # only number is valid unless v.numeric? - invalid_records.push record_index - warn_stat "invalid stats value: #{v}", monitor - - exit 1 + result[k] = v + next end v = v.index('.') ? v.to_f : v.to_i @@ -84,6 +82,8 @@ min_cols_stat = '' max_cols_stat = '' zero_stats = [] result.each do |key, val| + next if val.is_a?(String) + if max_cols < val.size max_cols = val.size max_cols_stat = key -- 2.23.0

1 1

[PATCH v2 compass-ci 2/3] libvirt/lib: introduce mylog.rb
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in the fulture [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 in this way, user can perceived message when error occurred Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..141dd80 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + @name = filename + rm_exist? + super(filename) + format + end + + private + + def rm_exist? + if FileTest.exist?(@name) + File.delete(@name) + end + end + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: set default values
by Liu Yinsi 22 Jan '21

22 Jan '21

[why] when call simplify-ci individually to deploy environment, SCHED_HOST, SCHED_PORT, GIT_SERVER, these variables are undefined in a very clean machine, execute the script will failed, so set default values is necessary. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f772..df18272f3 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,18 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${SCHED_HOST:=172.17.0.1} +: ${SCHED_PORT:=3000} + git_ci() { + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi + git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 22 Jan '21

22 Jan '21

the maximum time for borrowing machine should no more than 10 days. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..e9e693c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -64,6 +64,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + check_run_time() set_sshr_info() return end @@ -410,6 +411,19 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash["testcase"] == "borrow" + + if @hash["runtime"].ends_with? "d" + hour = hash["runtime"][0..-2].to_i * 24 + elsif @hash["runtime"].ends_with? "h" + hour = hash["runtime"][0..-2].to_i + end + + error_msg = "The maximum runtime should no more than 10 days." + raise error_msg if hour.is_a?(Int32) && hour >= 240 + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb support merge 'message' to stats
by Lu Weitao 22 Jan '21

22 Jan '21

[Example] { "iperf.tcp.sender.bps": 34804801216.197174, "iperf.tcp.receiver.bps": 34804762215.18231, "iperf.iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..5920d62 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -10,6 +10,8 @@ require "#{LKP_SRC}/lib/matrix" require_relative './params_group.rb' def set_pre_value(item, value, sample_size) + return value if value.is_a?(String) + if value.size == 1 value[0] elsif independent_counter? item -- 2.23.0

1 0