[PATCH v2 compass-ci 1/6] git_mirror: add webhook handler
webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing Signed-off-by: Li Yuanchao <lyc163mail@163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0
+# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end
blank line here. Thanks, Jiaxin
+ # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2]
On Fri, Nov 06, 2020 at 09:22:10AM +0800, Lin Jiaxin wrote:
+# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end
blank line here.
Yes, a blank here makes it looks better. Thanks, Yuanchao
Thanks, Jiaxin
+ # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2]
On Thu, Nov 05, 2020 at 08:13:00PM +0800, Li Yuanchao wrote:
webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing
Signed-off-by: Li Yuanchao <lyc163mail@163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end
def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2]
fork_name, project = webhook_url.split('/')[-2, 2]
+ + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url)
the project is unique, how about just use project name to find the 'git_repo' from the @fit_repo instead of the three tries Thanks, Luan Shengde
+ + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0
On Fri, Nov 06, 2020 at 09:46:21AM +0800, Luan Shengde wrote:
On Thu, Nov 05, 2020 at 08:13:00PM +0800, Li Yuanchao wrote:
webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing
Signed-off-by: Li Yuanchao <lyc163mail@163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end
def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2]
fork_name, project = webhook_url.split('/')[-2, 2]
ok
+ + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url)
the project is unique, how about just use project name to find the 'git_repo' from the @fit_repo instead of the three tries
A project may have many forks, like linux, many people fork it and have their own linux project. If more than one of them in our compass-ci platform, search project name is not enough Thanks, Yuanchao
On Thu, Nov 05, 2020 at 08:13:00PM +0800, Li Yuanchao wrote:
webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing
Signed-off-by: Li Yuanchao <lyc163mail@163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end
def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url)
What it check? Please give more descriptive function name. Thanks, Fengguang
+ return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0
participants (4)
-
Li Yuanchao -
Lin Jiaxin -
Luan Shengde -
Wu Fengguang