- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 1/2] sbin/submit: add submit_id
by Wu Zhende 25 Jan '21

25 Jan '21

A globally "submit_id" is automatically generated for each submit command and automatically added to jobs. Indicates the jobs of the same batch submitted by the command. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/submit b/sbin/submit index a92556fff..5273896b6 100755 --- a/sbin/submit +++ b/sbin/submit @@ -103,6 +103,9 @@ def find_jobfile(jobfile) exit 1 end +submit_id = %x(uuidgen).chomp +puts "submit_id=#{submit_id}" + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -113,6 +116,7 @@ ARGV.each do |jobfile| jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') jobs['node_roles'] ||= 'server client' if jobs['cluster'] + jobs['submit_id'] = submit_id jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] -- 2.23.0

1 0

[PATCH compass-ci] taskqueue: add start dependent container: redis
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] The container taskqueue start depends on redis, so add redis to taskqueue/start-depends. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/taskqueue/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/taskqueue/start-depends diff --git a/container/taskqueue/start-depends b/container/taskqueue/start-depends new file mode 100755 index 000000000000..7800f0fad3ff --- /dev/null +++ b/container/taskqueue/start-depends @@ -0,0 +1 @@ +redis -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/dump-stats: keep message line
by Lu Weitao 25 Jan '21

25 Jan '21

[Example] { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ], "iperf.run.message": "error xxx" } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/dump-stat | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sbin/dump-stat b/sbin/dump-stat index b5abf5a9b..5d565e36a 100755 --- a/sbin/dump-stat +++ b/sbin/dump-stat @@ -59,6 +59,12 @@ while (line = STDIN.gets) exit 1 end + + if k.end_with?('message') + result[k] = v + next + end + # only number is valid unless v.numeric? invalid_records.push record_index @@ -84,6 +90,8 @@ min_cols_stat = '' max_cols_stat = '' zero_stats = [] result.each do |key, val| + next if val.is_a?(String) + if max_cols < val.size max_cols = val.size max_cols_stat = key -- 2.23.0

2 2

[PATCH v4 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 25 Jan '21

25 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

2 2

[PATCH compass-ci] taskqueue: add build dependent container: redis
by Yu Chuan 25 Jan '21

25 Jan '21

[Why] The container taskqueue depends on redis, so add redis to taskqueue/build-depends. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/taskqueue/build-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/taskqueue/build-depends b/container/taskqueue/build-depends index 5dd403e166af..a961b878fb44 100755 --- a/container/taskqueue/build-depends +++ b/container/taskqueue/build-depends @@ -1 +1,2 @@ scheduler-dev +redis -- 2.23.0

2 4

[PATCH v4 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 23 Jan '21

23 Jan '21

request url: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

2 1

[PATCH v4 compass-ci 2/3] libvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in this way, user can get message when error occurred [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin/quick_fetch: add an interface script for git_mirror speed up
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/quick_fetch | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 sbin/quick_fetch diff --git a/sbin/quick_fetch b/sbin/quick_fetch new file mode 100755 index 0000000..e8378ec --- /dev/null +++ b/sbin/quick_fetch @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +# usage: quick_fetch git_repo:$git_repo +# or: quick_fetch $url +# quick_fetch https://github.com/Siguyi/AvxToNeon +# quick_fetch git_repo:u/upstream-repos/upstream-repos +def send_url_of_repo + connection = Bunny.new('amqp://172.17.0.1:5672') + connection.start + channel = connection.create_channel + + queue = channel.queue('web_hook') + message = ARGV[0] + queue.publish(message) + connection.close +end + +send_url_of_repo -- 2.23.0

1 0

[PATCH compass-ci 1/2] git_mirror.rb: add one more parameter choice when use webhook
by Li Yuanchao 23 Jan '21

23 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6d1624f..a111e51 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -318,6 +318,8 @@ class MirrorMain # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) + return webhook_url.split(':')[1].gsub(' ', '') if webhook_url =~ /^(git_repo:)/ + fork_name, project = webhook_url.split('/')[-2, 2] git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] bvirt/lib: introduce mylog.rb
by Xiao Shenwei 23 Jan '21

23 Jan '21

[why] records run logs 1. easy debugging 2. the log will be upload to result_root in the fulture [how] custom format of log like this: 2021-01-22 10:11:15 +0800 INFO -- SCHED_HOST: 3000 in this way, user can perceived message when error occurred Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..38f8c98 --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + File.delete(filename) if File.exist?(filename) + super(filename) + format + end + + private + + def format + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _progname, msg| + msg = if msg.is_a? Exception + ["#{msg.backtrace.first}: #{msg.message.split("\n").first} (#{msg.class.name})", + msg.backtrace[1..-1].map { |m| "\tfrom #{m}" }].flatten + else + msg.to_s.split("\n") + end + msg.map { |m| "#{datetime} #{severity} -- #{m}\n" }.join + end + end +end -- 2.23.0

2 2