February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] container/dracut-initrd: add --fix-missing when apt-get install
by Xu Xijian 04 Feb '21

04 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 7e2bf51..75148a1 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -5,9 +5,9 @@ export DEBIAN_FRONTEND=noninteractive apt-get update -apt-get install -y --no-install-recommends apt-utils >/dev/null 2>&1 -apt-get install -y nfs-common netbase cifs-utils kmod rsync -apt-get install -y dracut dracut-network dracut-config-generic +apt-get install -y --fix-missing --no-install-recommends apt-utils >/dev/null 2>&1 +apt-get install -y --fix-missing nfs-common netbase cifs-utils kmod rsync +apt-get install -y --fix-missing dracut dracut-network dracut-config-generic apt-get clean rm -rf /var/lib/apt/lists/* -- 2.23.0

2 2

[PATCH compass-ci] git_mirror.rb: fix that git clone failure isn't known
by Li Yuanchao 04 Feb '21

04 Feb '21

Some repo git clone failed just puts "Cloning into '/srv/git/t/twitch/twitch.git'..." and nothing more. So can not take "stderr.include?('fatal')" as the condition to judge fail or not. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8898a44..940d5e2 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -39,14 +39,12 @@ class GitMirror end def git_clone(url, mirror_dir) - ret = false url = get_url(Array(url)[0]) 10.times do - stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) - ret = !stderr.include?('fatal') - break if ret + %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") end - return ret + return false end def git_fetch(mirror_dir) -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: puts the info about the define_file
by Li Ping 04 Feb '21

04 Feb '21

[why] test the function "submit -a" always need to check define_files in job.sh puts the info about the define_files when submit job.yaml is better. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 9174cadc7..f19d1e6dc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -680,6 +680,7 @@ class Job Dir.glob("#{LKP_SRC}/{*,*/*}/{#{program},#{program}-dev,#{program}.*}{,/*}").each do |file| file = File.realpath(file) if git_files_list.include?(file) + puts "define_file: #{file}" relative_path = file.sub("#{LKP_SRC}/", '') @job["define_files"][relative_path] = File.read(file) end -- 2.23.0

1 0

[PATCH v2 compass-ci] container/redis: delete rebundant code
by Liu Yinsi 04 Feb '21

04 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/redis/start | 5 ----- 1 file changed, 5 deletions(-) diff --git a/container/redis/start b/container/redis/start index 34266a0..5e2dc44 100755 --- a/container/redis/start +++ b/container/redis/start @@ -22,8 +22,3 @@ cmd=( ) "${cmd[@]}" - -# test -echo you can use cmd: redis-cli to start a redis client -echo and then use cmd: keys * - -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: optimize the use of hash.merge
by Li Yuanchao 04 Feb '21

04 Feb '21

1. hash_c = hash_a.merge(hash_b) will return a new hash which contains key-values of both hash_a and hash_b, and if the two hashes have some same keys, the key-values of hash_b will be kept in hash_c. So that if a repo file has the same key with DEFAULTS file, its own will be kept. 2. fix that if there's no DEFAULTS file, the merge_defaults method will return nil. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index fa2e0cc..8898a44 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -453,10 +453,12 @@ class MirrorMain def merge_defaults(git_repo) defaults_key = File.dirname(git_repo) while defaults_key != '.' - return @git_info[git_repo].merge!(@defaults[defaults_key]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @git_info[git_repo].merge!(@defaults['default']) if @defaults['default'] + return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + + return @git_info[git_repo] end end -- 2.23.0

2 1

[PATCH compass-ci] redis: fix PoolTimeoutError
by Wu Zhende 04 Feb '21

04 Feb '21

[Error] Exception: No free connection (used 64 of 64) after timeout of 0.01s (Redis::PoolTimeoutError) [How] I looked at the number of concurrent Redis visits on our cluster, and it peaked at over 900. The scheduler and TaskQueue are about 50/50. So increase the size of the Redis connection pool. Adapt to our concurrency Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/redis_client.cr | 9 +++++++-- src/taskqueue/constants.cr | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b90759f..dfec365 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -32,6 +32,7 @@ class Sched def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new + Redis::Client.set_pool_size(1000) @redis = Redis::Client.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index a8d9dcb..030d1b7 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -12,13 +12,18 @@ class Redis::Client class_property :client HOST = (ENV.has_key?("REDIS_HOST") ? ENV["REDIS_HOST"] : JOB_REDIS_HOST) PORT = (ENV.has_key?("REDIS_PORT") ? ENV["REDIS_PORT"] : JOB_REDIS_PORT).to_i32 + @@size = 25 def self.instance Singleton::Of(self).instance end - def initialize(host = HOST, port = PORT) - @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) + def initialize(host = HOST, port = PORT, pool_size = @@size) + @client = Redis::PooledClient.new(host: host, port: port, pool_size: pool_size, pool_timeout: 0.01) + end + + def self.set_pool_size(pool_size) + @@size = pool_size end def keys(pattern) diff --git a/src/taskqueue/constants.cr b/src/taskqueue/constants.cr index c702382..6240c54 100644 --- a/src/taskqueue/constants.cr +++ b/src/taskqueue/constants.cr @@ -11,7 +11,7 @@ REDIS_PORT = 6379 # delimiter and exttract-ststs will loop consume job # when use 32 (scheduler use 25), meet Exception: # Exception: No free connection (used 32 of 32) -REDIS_POOL_NUM = 64 +REDIS_POOL_NUM = 1000 REDIS_POOL_TIMEOUT = 10 # ms -- 2.23.0

1 0

[PATCH lkp-tests] tests/monitor-job: fix lkp-tests gem depend
by Wei Jihui 04 Feb '21

04 Feb '21

faye-websocket is needed, but elasticserch is not needed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/monitor-job | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/monitor-job b/tests/monitor-job index 0f495ae2..5da992fa 100755 --- a/tests/monitor-job +++ b/tests/monitor-job @@ -19,7 +19,7 @@ set_scheduler_config() install_gem_pkgs() { gem sources -r http://rubygems.org -a https://gems.ruby-china.com/ - gem install activesupport git rest-client elasticsearch + gem install git activesupport rest-client faye-websocket } build_my_info() -- 2.23.0

1 0

[PATCH v2 compass-ci] container/register-accounts: fix no my_token
by Luan Shengde 04 Feb '21

04 Feb '21

the lib/build_my_info_client will not generate my_token automatically. and lib/build_my_info_client do not support update if account exists in the ES. use lib/build-my-info, it will generate my_token if it's not in my_info. fix rubocop warning: register-accounts:11:18: W: Lint/UnusedBlockArgument: Unused block argument - key. If it's necessary, use _ or _key as an argument name to indicate that it won't be used. content.each do |key, value| ^^^ Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/register-accounts/register-accounts | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/register-accounts/register-accounts b/container/register-accounts/register-accounts index 519f289..e4d39ff 100755 --- a/container/register-accounts/register-accounts +++ b/container/register-accounts/register-accounts @@ -5,16 +5,15 @@ require 'yaml' require_relative "#{ENV['CCI_SRC']}/lib/es_client" -require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" +require_relative "#{ENV['CCI_SRC']}/sbin/build-my-info" content = YAML.load_file('/etc/compass-ci/register/register.yaml') -content.each do |key, value| +content.each do |_key, value| my_info = {} my_info['my_email'] = value['my_email'] my_info['my_name'] = value['my_name'] my_info['lab'] = ENV['lab'] - build_my_info = BuildMyInfo.new(my_info['my_email']) - build_my_info.store_account_info(my_info) + build_my_info(my_info) end sleep -- 2.23.0

1 0

[PATCH compass-ci] submit/Dockerfile: better way to replace gem source
by Cui Yili 04 Feb '21

04 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/submit/Dockerfile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile index 3ca13f4..6a021d4 100644 --- a/container/submit/Dockerfile +++ b/container/submit/Dockerfile @@ -16,8 +16,7 @@ RUN apt-get update && \ # it may take a long time, even break up if you use default source address for the gem. # delete the following step if you do not want to use the default one. -RUN gem sources --add https://gems.ruby-china.com/ && \ - gem sources --remove https://rubygems.org/ +RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN gem install git faye-websocket activesupport rest-client -- 2.23.0

1 0

[PATCH compass-ci] providers/libvirt: consume job by libvirt
by Xiao Shenwei 04 Feb '21

04 Feb '21

function: 1. check host is the capability of executing tasks? - check libvirt - check qemu-kvm 2. request boot.libvirt and load files based on response 3. generate a domain.xml and start a vm used ruby-libvirt Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 68 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index a500c31..2883197 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -9,6 +9,8 @@ require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" +require_relative "#{ENV['CCI_SRC']}/providers/lib/domain" +require_relative "#{ENV['CCI_SRC']}/providers/lib/libvirt" def create_logger(hostname) filename = "#{hostname}.log" @@ -48,17 +50,75 @@ def request_job(context, sched_client, logger) return response end +def parse_response(response, context, resource) + context.merge!(response) + resource.parse_response(response) + context.merge!(resource.info) +end + +def generate_domain(context, logger) + domain = Domain.new(context, logger) + xml_path = '' + begin + domain.generate + xml_path = domain.save_to('domain.xml') + rescue StandardError => e + logger.error(e) + logger.error('please check your xml file format') + ensure + upload_log(context) + end + return xml_path +end + +def upload_log(context, upload_qemu_log: false) + id = context.info['job_id'] + host = context.info['LKP_SERVER'] + port = context.info['RESULT_WEBDAV_PORT'] + result_root = context.info['result_root'] + result_url = "http://#{host}:#{port}#{result_root}" + log_name = "#{context.info['hostname']}.log" + system "curl -sSf -T #{log_name} #{result_url}/ --cookie 'JOBID= #{id}'" + return unless upload_qemu_log + + qemu_log_name = "/var/log/libvirt/qemu/#{id}.log" + return unless FileTest.exist?(qemu_log_name) + + system "curl -sSf -T #{qemu_log_name} #{result_url}/ --cookie 'JOBID= #{id}'" +end + +def start(context, logger, libvirt) + xml_path = generate_domain(context, logger) + return if xml_path.empty? + + begin + libvirt.define(xml_path) + libvirt.create + libvirt.wait + rescue StandardError => e + logger.error(e) + ensure + upload_log(context, upload_qemu_log: true) + end +end + +def clean(context, sched_client, libvirt) + sched_client.delete_mac2host(context.info['mac']) + sched_client.delete_host2queues(context.info['hostname']) + libvirt.close +end + def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + libvirt = LibvirtConnect.new resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? - resource.parse_response(response) - context.merge!(resource.info) - sched_client.delete_mac2host(mac) - sched_client.delete_host2queues(hostname) + parse_response(response, context, resource) + start(context, logger, libvirt) + clean(context, sched_client, libvirt) end -- 2.23.0

1 0