January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container/add-software: add to clean hidden files in runtime dirs
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the first pkgbuild_repo if no '-git'
by Li Yuanchao 12 Jan '21

12 Jan '21

As new pkgbuild_repos don't have '-git' at the end, return the first one instead. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 8845b13..d765094 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -18,7 +18,7 @@ class AutoSubmit end def get_pkgbuild_repo(repo_array) - pkgbuild_repo = nil + pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| next unless repo =~ /-git$/ -- 2.23.0

2 1

[PATCH compass-ci] container: mount default config dir to delimiter container for submitting job
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/delimiter/start b/container/delimiter/start index 27e8d45..5d7037d 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -23,6 +23,7 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm 'delimiter' cmd = %w[ @@ -34,8 +35,9 @@ cmd = %w[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} - -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter -- 2.23.0

2 1

[PATCH v2 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 12 Jan '21

12 Jan '21

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..4ff4285 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,32 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +98,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

2 1

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 12 Jan '21

12 Jan '21

>+ def lftp_mirror >+ @result_root.delete_prefix!('/srv') >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' >+ url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" >+ system "lftp -c mirror #{url} >/dev/null 2>&1" >+ end >+ >+ def lftp(data) >+ if @result_root >+ lftp_mirror Seems it'll repeatedly call lftp on every new message? When do we start downloading? Ideally do it once after extract stats. >+ else >+ set_result_root(data) That else seems illogical. Typically we set a var before checking it. Maybe lftp_mirror will be called at the next message, however it's still illogical thing. Thanks, Fengguang >+ end >+ end >+ > def stop(data, web_socket, code = 1000, reason = 'normal') > @stop_query.each do |key, value| > return false unless data[key] == value >@@ -109,7 +136,7 @@ class Monitor > > @query.each do |k, v| > @query[k] = JSON.parse(v) >- rescue >+ rescue StandardError > end > query = @query.to_json > puts "query=>#{query}" >@@ -136,6 +163,7 @@ class Monitor > > output(data) > connect(data, ws) >+ lftp(data) > > stop(data, ws) if @action['stop'] > end >diff --git a/sbin/submit b/sbin/submit >index 05b3e86dd..a92556fff 100755 >--- a/sbin/submit >+++ b/sbin/submit >@@ -47,6 +47,10 @@ options = OptionParser.new do |opts| > actions << 'connect' > end > >+ opts.on('-l', '--lftp', 'lftp mirror job result') do >+ actions << 'lftp_result' >+ end >+ > opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do > opt_monitor = true > filter = YAML.load ARGV[0] if ARGV[0] >-- >2.23.0 >

1 0

[PATCH compass-ci] lib/job.cr: set sshr info when submit reproduce job
by Wu Zhende 12 Jan '21

12 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 6f344fe..218e2e1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -62,6 +62,7 @@ class Job return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") check_account_info() + set_sshr_info() return end end @@ -203,7 +204,7 @@ class Job end private def set_sshr_info - return unless @hash["ssh_pub_key"]? + return unless @hash.has_key?("ssh_pub_key") self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] -- 2.23.0

2 1

[PATCH lkp-tests 1/3] stats/crystal/memory--stat.cr: create memory--stat.cr with memory--stat
by Lu Kaiyi 12 Jan '21

12 Jan '21

create memory--stat.cr as same content with memory--stat Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/memory--stat.cr diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr new file mode 100644 index 000000000..1a39e4075 --- /dev/null +++ b/stats/crystal/memory--stat.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby + +# time: 123456 +# cgroup: pmbench.1 +# cache 0 + +def parse + cg = 'nocg' + $stdin.each_line do |line| + case line + when /^time:/ + puts line + when /^cgroup: (\S+)/ + cg = $1 + when /(\S+)\s*(\S+)/ + k, v, = line.split + puts "#{cg}.#{k}: #{v}" + end + end +end + +parse -- 2.23.0

2 1

[PATCH v3 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..e3dfd17 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo SCHED_PORT: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 0

[PATCH v2 compass-ci] my-qemu: make output better
by Ren Wen 12 Jan '21

12 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..6acfc2e 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -74,7 +74,7 @@ get_initrd() print_message() { - echo $SCHED_PORT + echo sched_port: $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append -- 2.23.0

1 1

[PATCH compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..03bb061 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,16 @@ run_docker() done } +kill_process() +{ + multi_qemu=`ps -ef |grep multi-qemu |grep "$USER" |awk '{print $2}'` + multi_docker=`ps -ef |grep multi-docker |grep "$USER" |awk '{print $2}'` + kill -9 $multi_qemu + kill -9 $multi_docker +} + run_qemu run_docker wait + +kill_process -- 2.23.0

3 2