When using Jenkins with GitHub, it is now possible to create something called a 'GitHub org folder,' which is a 'smart folder' that monitors your entire GitHub org. Anytime a new repo is created in GitHub, with a branch that has a Jenkinsfile in it, Jenkins automatically creates a new Pipeline multibranch job and sets up appropriate triggers. This used to be called GitHub Org Folder, it's now called GitHub Branch Source.
@kohsuke has written a new plugin which depends on gitlab-plugin and provides this functionality: https://github.com/kohsuke/gitlab-branch-source-plugin But maybe we should consider supporting it natively in this plugin. It's not clear to me how much more work would be required to do that.
@trance1st @colourmeamused do you want to take a look at this and see what you think? It's beyond my ability, but I know there is a lot of excitement about GitHub Org Folder, and it would be good if we can try to keep parity with things that are supported with Jenkins + GitHub.
The plugin Owen linked implements everything that's necessary to make "GitLab org folder" work already. You can try it with mvn hpi:run.
What I'd like you guys to consider is whether we can "merge" two plugins by taking code in that repo and putting it here. From users' perspective I think it makes the most sense, because only a handful people in the world would be able to see the name "gitlab branch source plugin" and understand what it adds to your Jenkins.
Owen told me that this plugin used to use the java-gitlab-api that I'm using in gitlab-branch-source-plugin but since then moved away from it. I explained to him that my priority is to get this functionality delivered to users as fast as possible, and I don't have any attachment to the code. So if you decide to eliminate that dependency over time I'm perfectly happy, but I'm not very keen to doing that myself. I can imagine things like this might cause some hesitation to merge plugins, in which case I'm happy to keep the gitlab-branch-source plugin separate.
What I think is a great tragedy is for this plugin to "natively support it" by essentially reimplementing the same feature in a different way, which results in duplicated functionality that spreads us even thinner.
@coder-hugo if you have a couple of minutes, can you refresh my memory about why we decided to stop using the java-gitlab-api library, and instead just implement our own access to GitLab? IIRC there was a bug or missing functionality in that library, and you concluded that it was pretty easy to just call it directly ourselves.
Hi there,
i've begun to write a gitlab-branch-source-plugin inspired by the github-branch-source too - sorry, i started some weeks ago and then did not find kohsuke's plugin before or else i would have written earlier.
Just stumpled over this while looking for someone to contact about the same intent - wheter and how to merge the functionalities of both plugins as i am building on the gitlab-plugin and the gitlab-java-wrapper library too and would like to find a way to reduce duplication.
This is my first jenkins plugin and it's focus was on the use cases which are important to our company-internal-dev-team and for me to learn how to write a plugin for jenkins. Meaning: the code-quality is not really great and sometimes i just use "brute force" or code duplication to integrate functionality of the gitlab-plugin into my gitlab-source-plugin (e.g when accessing the cause-data).
Perhaps you guys want to take a look and then we'll see if there is a way to integrate the plugins or some such thing? https://github.com/Argelbargel/gitlab-branch-source-plugin
Kind regards,
Argelbargel
@Argelbargel nice work getting started writing a plugin!
At a minimum, I would recommend that you use @kohsuke's work as the basis for your solution. That is, if you can make improvements to that codebase, please do that, rather than releasing another plugin that does the same thing. As a community we are trying to reduce the amount of duplication in the plugin ecosystem whenever possible.
In terms of integrating this functionality with the gitlab-plugin, to me it does seem like a logical thing to do. It's just that I do not have the Java experience to do it myself. That's why I was trying to get @trance1st or @colourmeamused to weigh in, since they are the Java-capable maintainers of this plugin :)
Wow, @Argelbargel it's a great work! I just quickly skimmed over the code but I think your version looks more feature rich than my version. You've already good some hook support, icon, and so on.
I also got in touch with people from GitLab who's interested in helping. I'll write to you in email to figure out how to consolidate the effort and make introduction.
In the mean time, @omehegan, we haven't heard one way or the other from your other maintainers. Given their silence, I'd like to suggest that we merge gitlab-branch-source-plugin (of some flavor TBD now that we have two codebases) into gitlab-plugin and add relevant people to the maintainers. WDYT?
As discussed with @Argelbargel and @omehegan , the merged effort is in https://github.com/jenkinsci/gitlab-plugin/tree/branch-source
Right now, I just joined two codebase and made it to compile.
The plan is for @Argelbargel to lead the effort to massage the branch source part further so that it can sit nicely together with the rest of the gitlab plugin. I'm going to pull in a few more people I've been talking to with regard to this effort.
Hi guys,
What is the current status of this feature?
It would make our work easier, so I'd really like to use this! :)
Hi @TYsewyn ,
i' m really sorry, but up until now there did not happen a lot. I've been swamped with other work and thus just did some bug-fixes over at https://github.com/Argelbargel/gitlab-branch-source-plugin but did not really work to integrate it's functionality into the gitlab-plugin itself.
My own plan currently is to first make my plugin work with pipeline multibranch projects because i think the required changes will make integration of the plugins somewhat (while not that much) easier and try again to contact @omehegan and the other maintainers to get the integration done.
Should the maintainers of the gitlab-plugin like to start with the integration or have some ideas on how to get it done, i'm keen to help though ;-)
@mreichel and @colourmeamused feel free to comment on this if you are interested in helping with this integration. We also have an offer of support from @stephenc, a longtime Jenkins contributor and plugin author, if we need guidance about the implementation.
If you have a rough idea what still needs to be done I can help out with that too.
Basically you want to have the same functionality like the Github plugin, and extend this with the following:
When the proper webhooks are configured, this plugin automatically detects new projects, branches and merge requests, adds new jobs accordingly and builds them. If Jenkins (or rather the user configured to communicate with GitLab) is allowed to do so, it will setup these webhooks automatically for you.
When Jenkins is allowed to do so build results will be reported back to GitLab. You can choose wheter only the build result (success/failure) get reported or if Jenkins reports every stage defined in your pipeline.
I'm not sure how you want to implement the creation/deletion of the webhooks in GitLab, but then again I didn't take the time to look at the code. :)
The first problem i ran into when thinking about how to integrate them is this: as the original plugin is basically a build-trigger for a single project it simply uses the full path to the project to identify which project to trigger when called by a webhook.
The gitlab-branch-source-plugin uses the branch- and scm-apis to detect which projects/branches to build. The scm-api defines an event-system which requires that scm-sources are globally identifiable. So when we have two different jobs created by the gitlab-branch-source-plugin which point to the same gitlab-project/branch and there is, let's say a push to that branch, both jobs must be triggered. Thus the webhook-urls of the gitlab-branch-plugin use a checksum over some of the sources data to uniquely identify them.
So the first step should be to find a way to unify the url-schemes of both plugins to support both the build-trigger and the scm-api-based implementation while registering as few webhooks in gitlab as possible. As the gitlab-branch-plugin tries to do most of the webhook-management automatically (e.g. registering/deleting of hooks in gitlab), we should be able to detect whether a given webhooks is still required because there still exist projects for it. I could not find an easy we to satisfy those requirements and keep compatiblity with current installation.
But if/when we have a unified url-scheme in place we'd have to adapt the RootAction which accepts the webhook-requests so that it triggers both the "old" projects and the orginisation-folders or multibranch-pipelines.
The next thing to do would be to find another way to store the gitlab-connection than in a project-property as project-properties are quite error-prone to use in pipeline-projects (i currently use an action for that as of the latest release).
I really recommend waiting until I get the GitHub and Bitbucket UI refactoring complete:
Sounds great. Then i'll wait until the new API is released, update the gitlab-branch-plugin to that and then we take it from there...
Thanks guys, this sounds like a good plan of action.
Very good, count me in once the new release is done
@stephenc just wanted to check and see what the status of your refactoring work is. I haven't seen any new posts to the thread on the jenkins-dev list since June, so I presume you have had other things taking priority.
All done. Released yesterday. Recommended example reference impl is gitea plugin
@Argelbargel I'm at Jenkins World this week, and various folks have asked me about progress on this functionality. Do you think you will have time to work on it in the near future?
@omehegan i hope, i'll find some more time to add least go some little steps to implementing the branch-source-functionality into the gitlab-plugin. after some digging through the code-bases, i think that simply "merging" them is not going to work. These are steps i'd like to take:
Preferably i'd like to add stuff here while removing it from the gitlab-branch-source plugin.
@Argelbargel OK, this sounds good. Do you think it could be worthwhile for someone else to work on this with you? Maybe we could find someone else who is willing to contribute some time toward it.
Of course! I'll welcome any help i can get with this :-)
@mreichel @colourmeamused @christ66 if any of you would be interested in working on this feature with @Argelbargel, he is looking for help!
I am off for holidays, if @Argelbargel is still looking for help afterwards, I will be available
Really looking forward to https://github.com/Argelbargel/gitlab-branch-source-plugin getting merged into this.
This is dead?
Not yet - i've just been either moving between flats or been swamped by my day job :-/ I'm still hoping that we'll need some more features for the gitlab-branch-source-plugin at work so that i might sneak in some work on merging the plugins in may day work (but as always infrastructure work does not gain priority in the backlog easily) ;-) Or that my new flatmate and i run out of stuff to hang on walls and such on weekends. But after months of defering and delaying i'd understand if someone with time to spent would take over and be happy to help...
@christ66 ^
I know bumps are unappreciated, but I think quite a few people might benefit greatly from this integration. It's a pity to have the integration with GitLab not fully in the plugin yet. I tested the basic functionality of this plugin a while ago and it was working great so I've been hoping for the official integration of this for quite some time :)
The current maintainers just have limited time to work on this feature, because landing it will require a non-trivial amount of work. Offers of help coding would be appreciated :)
I would love to help once I have implemented the same for Accurev Plugin 👍 Hoping to have something working for Accurev during this month.
@casz I hope you are implementing trait support from the start and please use the pronoun support to ensure that the accurev terminology is correctly represented (e.g. streams not branches, snapshots not tags) as it would be super awesome to confirm that the pronoun stuff actually works as advertised and that I documented how to do it correctly (in https://github.com/jenkinsci/scm-api-plugin/blob/master/docs/implementation.adoc there is even a code snippet underneath the text "For example, with the Accurev source control system we might have" which should be showing exactly what you need to do 😉 )
Trait support is definitely coming!
I just want to get the Navigator, Source, SCM part working first so I can see scanning depot for streams working. Sadly I have to implement a lot of logic myself, I see limited reuse from Git Plugin.
I plan to borrow some of the traits from Git plugin and wrapping them in my own AccurevSCMTrait class 👍
What is the status of GitLab Branch Source and its merging into the GitLab Plugin?
It is my understanding that it will work much like the GitHub Branch Source. Would it also support sending build status back to GitLab, so that a merge request will show if its build has passed or not.
@Argelbargel do you think you will be able to work on this in the near future at all?
Hello,
Is that feature still considered to be developed and finally (hopefully) merged?
@ptrtonull we would definitely like to incorporate this feature, but AFAIK it is not being worked on right now, and I do not have capacity to complete it myself.
Hello!
We need this functionality very much! Is there any way we can help?
@serge2016 continued work on this seems to be stalled. If you are interested in trying to move it forward, I can help point you to the existing, incomplete solution, and the reference implementation that we should use as a guide. And I can coordinate testing and review efforts. Let me know if you are interested.
@omehegan Any news on the issue? I could help with implementation if you'd point me where to start.
@qnox in recent conversations with others, I think we have felt like the original idea of providing branch source support WITHIN this plugin is not the best way forward. Github and Bitbucket support for branch sources is handled in dedicated plugins, and I guess the potential for user confusion (needing two plugins to get a complete Gitlab integration) is outweighed by the much easier process of getting the integration to work when it is stand-alone.
Given that, I think the path forward would be to start from this partial implementation: https://github.com/Argelbargel/gitlab-branch-source-plugin I have not heard anything from @Argelbargel in over a year, so I assume he is not still working on it, although some PRs have been merged to it. Update it as necessary to use the SCM API changes referenced here: https://github.com/jenkinsci/gitlab-plugin/issues/499#issuecomment-303688183 And, rather than using the https://github.com/timols/java-gitlab-api library which we had some issues with in the past, consider using the GitLab API classes that are native to this plugin. It may be necessary to move those into a third helper plugin that this one and the branch source one both rely on. Otherwise, I'm just concerned that with two unrelated API implementations, we could get inconsistent behavior.
If you or anyone else wants to move this forward, I would welcome it!
Hi @omehegan, I am writing a proposal for GSoC '19 to address this issue. Since Jenkins does not have a Multibranch support for GitLab released yet, my work would be based on making a production ready GitLab-Branch-Source-Plugin. I would extend my help by updating the SCM API changes made by @stephenc and replacing java-gitlab-library with native GitLab API classes for the BS plugin by @Argelbargel. I also find that GitLab Plugin is also not actively maintained as you have mentioned in the docs. I also would like to extend my support for this repository as well. Would you reconsider your plan of having 2 separate plugins? I mean we can still try to build a unifying interface that handles discovering abstract git branches and pull requests (maybe we can extend the Branch API plugin) that provides all the APIs for the GitLab plugin to interact with single branch and multi branch at the same time. I want to know if I am thinking in the right direction.
I would properly be able to mentor this work since I have experience maintaining the bitbucket-branch-source-plugin and currently, actively use gitlab-branch-source-plugin made by argelbargel.
I would second having it in one plugin.
Suggest that the GitLab API is a separate plugin so that others can benefit for the GitLab API being accessible to other plugins.
There are hopes/plans to move the BitBucket API out of the bitbucket-branch-source-plugin so other plugins can benefit from the work.
Hi @casz good to know that you are willing to mentor this work. Let me ask in Jenkins gsoc sig channel if you can officially be a GSoC mentor. Regardless of that we can still collaborate and start working on this. Btw what you suggest is having single branch and multi branch features in one plugin and another plugin just to expose the GitLab APIs?
Yes, exactly one plugin for the SCM part (single and multi) and one plugin for GitLab API interaction.
For both plugins, I highly recommend looking at the usage of handyURI which is available as an API plugin. https://github.com/jenkinsci/handy-uri-templates-2-api-plugin
We use it in Bitbucket-branch-source-plugin and gitea-plugin to manage all URI endpoints it is heavily used here https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/80
@casz I was wondering how does BitBucket manage to merge the functionality of both single and multi branch builds while other SCM (Gitlab and Github) plugins have 2 separate plugins. How does BitBucket solve this issue which talks about the difficulty of merging the 2 plugins? Another thing I would like to know, as you suggest of having a BitBucket API, will it be similar to the GitHub API by @kohsuke? Also would it be a good idea to build a unifying interface common for all SCM(Gitlab, Github, Bitbucket) providers (considering most have very similar functionalities and dependencies). The client plugins for respective SCMs will simply use this unifiying interface api and their own SCM API to provide single and multibranch support?
In case you got confused, this is a bottom up approach of what I am proposing:
1) A unifying interface that wraps all the dependencies like scm api, branch api etc and implements some of the features that are common to all SCM plugins.
2) There will be separate SCM APIs for each SCM provider.
3) Then there will be separate client plugins that combined the above 2 plugins(SCM API and unifying interface) to provide single and multibranch build support.
This will enhance the understanding of the SCM plugins from user point of view because at the moment I feel there is no common point between the various SCM plugins.
I think both Github and Bitbucket (branch-source-plugins) relies on the git plugin to provide the single branch build support and only concerns themselves with multi branch 😃
Okay so the unifying interface thing exists as git-plugin, is that what you meant?
Repeating the first half of my earlier comment,
How does BitBucket manage to merge the functionality of both single and multi branch builds while other SCM (Gitlab and Github) plugins have 2 separate plugins? The BitBucket API, will it be similar to the GitHub API by @kohsuke?
@casz I was wondering how does BitBucket manage to merge the functionality of both single and multi branch builds while other SCM (Gitlab and Github) plugins have 2 separate plugins. How does BitBucket solve this issue which talks about the difficulty of merging the 2 plugins? Another thing I would like to know, as you suggest of having a BitBucket API, will it be similar to the GitHub API by @kohsuke?
I recommend reading https://github.com/jenkinsci/scm-api-plugin/blob/master/docs/implementation.adoc#naming-your-plugin as some of my snark in that section may illuminate the issue.
In general the single vs multibranch split is an accident of history. The recommended solution is to have one plugin provide both.
In the case of plugins that use Git as the underlying SCM, however, the reflection of user behaviour is to just let them use the Git plugin for single branch "old style" projects and the SCM provider specific plugin for multibranch.
I have been considering adding a "specialized" Git implementation for Gitea single-branch, but every time I hit the fact that there are crazy users who want to do crazy things with their single-branch builds, so you end up leaving them with Git anyway
@baymac I'm not sure how much of @Argelbargel's codebase you plan on leveraging, but me and a handful of others have made a number of fixes to the gitlab-branch-source plugin. Let me know if there's anything I can do to help. I think it also would be a good idea to open a discussion in the gitlab-branch-source-plugin repo so that everyone active over there is aware of your plan
@grfullerton Yes, I will do that very soon. I will sure take inspiration from the existing gitlab-branch-source plugin to add multibranch pipeline support. Thanks for letting me know. :)
@stephenc I need some suggestions from you. We are also planning to create a new plugin with name either Branch Source Plugin or Multibranch Plugin or SCM Traits Plugin which will contain the implementation of SCM traits that are common across all SCM providers (Github, Bitbucket, Gitlab, Gitea etc.). One use case of this plugin is:
Suppose a user just needs to index a multibranch project from multiple SCM providers and discover the branches, prs, tags etc (all common traits). You can find a pointer to such an application here. In this, they need to create a multi-branch pipeline on their platform. And they need to support github, gitlab, bitbucket multi-branch. Need is for a simple extension point of branch discover trait.
The benefit to the user is that he doesn't have to install multiple <SCM>-Branch-Source-Plugins for this purpose. Just one plugin will do the job.
I want to know if something of this type of plugin exists already? If not, is it a good idea to have this new plugin which can also be used by the <SCM>-Branch-Source-Plugins to use the common traits implementation and extend it for the implementation of exclusive traits?
PS - The whole idea is proposed by my mentor, I am requesting your suggestions.
CC: @LinuxSuRen
@grfullerton This gitlab-branch-source-plugin repo has done with most functions. And I try to install it to my Jenkins server. Just I hope I can use it as a multi-branch job instead of GitLab org folder. The idea of GitLab org folder is great. Just in my use case, I need another one.
@stephenc As @baymac said, I think that having a plugin to define the extension point for multi-branch trait (like how to discover PRs and how to trust the Jenkinsfile) will be better.
If I understand correctly, only git providers have the concept of PRs. If it's true. I would recommend that the new plugin's name is git-provider-api-plugin or something like that.
Any opinions are appreciated.
A “one plugin to rule them all” is a bad plan imho. Better is to add specialists SCM implementations that are tailored to their need. There should never have been a _GitHub Branch Source_ plugin, it should have been just _GitHub_ or _GitHub SCM_. It would be fine to have that plugin dependent on Git but instead providing a specialist SCM source that is adapted to GitHub needs (authentication, url conventions, etc)
The same should applie to the other hosting providers. Users just want to install the plugin for their hosting provider. A “multi” will just drown them in choices... which is bad
@stephenc I think what you are suggesting is between <SCM>-Plugin and <SCM>-BS-Plugin. At this moment we are not talking about that debate. Let's pause that debate for this discussion because community holds very contrasting views on that.
Let me explain what we are talking. Please bear with me.
All the Traits (different from a simple Git) implementation lies inside the <SCM>-BS-Plugins. We know that the Traits across the SCM Providers have many common implementation especially _PRs_ .
Example: Suppose one user just want to index a Multibranch project from multiple SCM Providers (Gitlab, Github, BitBucket) that detects branches, prs and tags. He does not need authentication or any additional behaviours/traits like the ones provided in <SCM>-BS-Plugins apart from PR Trait.
Issue: This cannot be done by a Git Plugin since it does not detect PRs. Now the only option he has is to install multiple <SCM>-BS-Plugins and add each branch sources which IMO is too large for the function he needs.
Idea: Develop a plugin which will implement the common traits of generic SCM Providers (Gitlab, Github, BitBucket etc) and optionally extend it to the <SCM>-BS-Plugins.
From user's perspective who wants to use their SCM Plugin will still face the <SCM> and <SCM>-BS dilemma. We are not talking about that at this moment. This is a specific use case which is different which a regular user would seldom require and possibly ignore altogether. We are talking about a new plugin which users only need when they have a very specific requirement like the one explained above example (simple index and detect PRs of their repository). Just to be clear this plugin does not add to the existing confusion.
My Questions:
Can this feature be infused inside the Git Plugin that it is able to detect PRs regardless of the SCM Provider? In a way that it detects the type of SCM Provider and allow to add 2 behaviours like 'Discover PR from origin' and 'Discover PR from fork'. Is this a bad idea?
If this plugin with the common SCM Provider Traits (especially PRs) is a bad idea then are you suggesting that the user installs multiple <SCM>-BS-Plugins and add branch source for each SCM even if it is for the purpose of indexing only (no auth, no wehooks, no behaviours configuration other than PR is required)?
If you want in the same multibranch project to detect PRs in multiple hosting providers, then yes you add multiple sources.
The original design was that you would have separate sources for PRs from regular branches, but the low GitHub rate limits forced the GitHub source to do multiple jobs.
Plus traits are per source so you’ll still need to have multiple sources anyway if you want PRs from both GitHub and Bitbucket (and you do t want that because some PRs will be “masked” by a matching PR number in a higher priority PR)
Also I don’t see PRs as exclusive to Git... for one, mercurial has PRs through Bitbucket Cloud.
Plus I think Perforce and Accurev may be capable of the equivalent concept (but probably under a different name: Change Requests)
Additionally, each provider has different subtlety about different aspects of PRs, so I don’t think there is enough commonality to merit the effort of having a common trait... far better would be to fix Symbol support so that all the traits can have the same symbol despite being different implementations
Thanks @stephenc for all the clarifications. It answered everything what I was looking for. Maybe @LinuxSuRen can have some questions.
Right now my knowledge of SCM API is limited, I'm working on that. I'll see what can be improved in the Symbol support soon. 😊
I still don't think that one git provider plugin provides all functions is a good idea. If the plugin is heavy, then it may cause more isuess. Such as security. That's just my own ideas. Don't let me block your process. @baymac
This functionality has been developed in a stand-alone plugin, available here: https://github.com/jenkinsci/gitlab-branch-source-plugin/
For those who have been patiently waiting for this for a long time, please install GitLab Branch Source and give it a try! If you encounter any problems, please work with the maintainers of that plugin to address them.
This functionality has been developed in a stand-alone plugin, available here: https://github.com/baymac/gitlab-branch-source-plugin
The plugin moved to:
https://github.com/jenkinsci/gitlab-branch-source-plugin/
Most helpful comment
Really looking forward to https://github.com/Argelbargel/gitlab-branch-source-plugin getting merged into this.