- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] git_mirror: feat: reload fork_info when /upstream-repos update
by Li Yuanchao 29 Sep '20

29 Sep '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index ad27880..88a94ac 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -147,6 +147,8 @@ class MirrorMain end def send_message(feedback_info) + feedback_info.merge!(@git_info[feedback_info[:git_repo]]) + feedback_info.delete(:cur_refs) message = feedback_info.to_json @message_queue.publish(message) end @@ -158,12 +160,12 @@ class MirrorMain @fork_stat[feedback_info[:git_repo]][:queued] = false return unless feedback_info[:possible_new_refs] + return reload_fork_info if feedback_info[:git_repo] == 'upstream-repos/upstream-repos' + new_refs = check_new_refs(feedback_info[:git_repo]) return if new_refs[:heads].empty? feedback_info[:new_refs] = new_refs - feedback_info.merge!(@git_info[feedback_info[:git_repo]]) - feedback_info.delete(:cur_refs) send_message(feedback_info) end @@ -219,4 +221,27 @@ class MirrorMain @git_info[git_repo][:cur_refs] = cur_refs return new_refs end + + def reload_fork_info + upstream_repos = 'upstream-repos/upstream-repos' + if @git_info[upstream_repos][:cur_refs].empty? + @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) + else + old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] + new_refs = check_new_refs(upstream_repos) + new_commit = new_refs[:heads]['refs/heads/master'] + changed_files = %x(git -C /srv/git/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + reload(changed_files) + end + end + + def reload(file_list) + system("git -C #{REPO_DIR} pull") + file_list.each_line do |file| + next if File.basename(file) == '.ignore' + + repo_dir = "#{REPO_DIR}/#{file}" + load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) + end + end end -- 2.23.0

2 1

[PATCH v4 compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) add field 'idle_job' for idle jobs, value is always 'true'. 2) add two fields 'FLUENTD_SERVER_*' for monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..309df04 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -468,9 +468,16 @@ class Sched def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" + extra_job_fields = [ + "idle_job=true", + "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", + ] + Jobfile::Operate.auto_submit_job( full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + "testbox: #{tbox_group}", + extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end private def add_kernel_console_param(arch_tmp) @@ -520,7 +527,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

1 0

[PATCH compass-ci] check path of the upstream_repo under /srv/git/
by Li Ping 29 Sep '20

29 Sep '20

--- lib/git.rb | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/lib/git.rb b/lib/git.rb index db97c47..a181ec9 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -3,8 +3,34 @@ # wrap common git commands class GitCommit def initialize(repo, commit) - @git_prefix = "git -C /srv/git/#{repo}.git" + @repo = repo @commit = commit + @git_prefix = cd_repo_path + end + + def check_repo + fork_name = @repo.split('/')[0] + project = @repo.split('/')[1] + first_letter = project[0] + fork_name = 'linus' if project == 'linux' + path_a = find_path("/srv/git/#{first_letter}/#{project}/#{project}.git") + path_b = find_path("/srv/git/#{first_letter}/#{project}/#{fork_name}.git") + path_c = find_path("/srv/git/#{project}/#{fork_name}.git") + path_d = find_path("/srv/git/#{project}/#{project}.git") + path_a || path_b || path_c || path_d + end + + def find_path(path) + return false if Dir.glob(path).empty? + + Dir.glob(path).first + end + + def cd_repo_path + find_repo = check_repo + return unless find_repo + + "git -C #{find_repo}" end def author_name -- 2.23.0

5 13

[PATCH v3 compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's filename is 'multi-qemu.yaml'. 3) add field 'idle_job' for idle jobs, value is always 'true'. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..6ca20d7 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,18 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + return nil unless Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.yaml"] + jobs.each do |job| + extra_job_fields = ["idle_job=true"] + if Path[job].basename == "multi-qemu.yaml" + extra_job_fields += ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +528,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

2 3

[PATCH compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's name is multi-qemu. job's name 'multi-qemu' is case-insensitive matching now. 3) `.yaml` and `.yml` can both match. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..c27dd99 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,20 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + if Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.y{a,}ml"] + extra_job_fields = ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + + jobs.each do |job| + if Path[job].basename =~ /^multi-qemu\.ya?ml/i + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}") + end + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +530,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

3 10

[PATCH compass-ci] check if author's email address exist in upstream repos
by Li Ping 29 Sep '20

29 Sep '20

--- lib/git.rb | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index a181ec9..4939c94 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,7 +2,7 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) + def initialize(repo, commit = nil) @repo = repo @commit = commit @git_prefix = cd_repo_path @@ -28,32 +28,42 @@ class GitCommit def cd_repo_path find_repo = check_repo - return unless find_repo + message = "The repo is not found in the upstream-repo" + + raise StandardError, message unless find_repo "git -C #{find_repo}" end + def check_author_in_upstream_repos(email) + author_email = %x(#{@git_prefix} log | grep '^Author'| grep #{email}).chomp + + message = "The author's email address is not found in the upstream-repo" + + raise StandardError, message if author_email.empty? + end + def author_name - `#{@git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}`.chomp + %x(#{git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}).chomp end def author_email - `#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}).chomp end def subject - `#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}).chomp end def commit_time - `#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}).chomp end def url - `#{@git_prefix} remote -v`.split[1] + %x(#{@git_prefix} remote -v).split[1] end def diff - `#{@git_prefix} diff -1 #{@commit}~..#{@commit}`.chomp + %x(#{@git_prefix} diff -1 #{@commit}~..#{@commit}).chomp end end -- 2.23.0

1 0

[PATCH compass-ci] sparrow: ignore containers when deploy
by Liu Yinsi 29 Sep '20

29 Sep '20

scheduler-https, conserver, web-backend-nginx no need to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..7698c08 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,7 +29,10 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit + [ "$container_name" == 'sub-fluentd' ] || \ + [ "$container_name" == 'conserver' ] || \ + [ "$container_name" == 'web-backend-nginx' ] || \ + [ "$container_name" == 'scheduler-https' ] && exit [ -x build ] && ./build [ -x install ] && ./install [ -x first-run ] && ./first-run -- 2.23.0

3 4

[PATCH v3 compass-ci 2/2] container/extract-stats: add 'LAB' in environment variables
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] src/extract-stats: fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

problem: stats_worker call add_task() of TaskQueueAPI to send delimiter task, but no task added in delimiter task queue. cause analysis: the latest TaskQueueAPI.add_task(service_queue_path, task), require {"lab" => $LAB} in task content, otherwise return error message: "409, need lab in task content" in fact, we miss "lab", so can not send task to task queue. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/extract-stats prefix set lab in ENV
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

2 5