Gitlab-plugin: Pipeline Multibranch builds are not supported

[omehegan]

Version 1.2.2 of the plugin currently supports regular Pipeline projects correctly, but does not support Pipeline Multibranch projects (see https://jenkins.io/doc/pipeline/#creating-multibranch-pipelines). See https://issues.jenkins-ci.org/browse/JENKINS-34396 for @jglick's suggestion of how to resolve this.

[coder-hugo]

@omehegan during the last days I played with the mulitbranch pipeline jobs and it looks like can't add "full support" for these. Our plugin can notify such jobs for new commits and send the build status back to GitLab and nothing more. The reason for the is how the multibranch pipeline jobs are working:
The multibranch pipeline job is just a SCMSourceOwner that can be notified about changes of the SCM. If this happens the multibranch pipeline job checks the configured SCMSource for changes and add new sub-projects for any branch that contains a Jenkinsfile or triggers a new build if the sub-project for the changed branch already exists. So we have no chance to pass some additional information (the gitlab webhook data) to the build. This means that for multibranch pipeline jobs none of our environment variables will be available.

[omehegan]

@coder-hugo thanks for looking into it! If the issue is just one of getting the GitLab parameters, have you looked to see how the Github plugin is handling this? I'm certain they have this support working.

@jglick any thoughts on this?

[coder-hugo]

@omehegan which Github plugin are you talking about? After a short look to the documentation of the GitHub plugin I haven't seen that this plugin provides any environment variable about the GitHub push event.

[omehegan]

@coder-hugo that's the one I mean. It says it provides trigger support by interpreting POSTs from Github, so I assumed that was implemented similarly to how we do it. Also that plugin's maintainer has joked that the original design of ours was just a copy-paste of that :)

[omehegan]

Oh, ok I guess I understand better now. To implement multibranch basically you just trigger the job and let the Git plugin figure things out in terms of which branch changed, etc... And so then we can't capture the other metadata from the hook, such as repo namespace and all that other stuff. But I guess if someone can live without those env vars when using this type of job, we could make the trigger work?

[coder-hugo]

@omehegan Yes our plugin can definitively trigger such jobs but without the metadata from GitLab. That's what I tried to explain in my first comment :wink:

[olhado]

Any chance this will be released soon? This would really accelerate adoption of the pipelines and the gitlab plugin where I work.

[gtudan]

Would be great to see this feature. Especially the commit-Status would be great. While you can use the step in multi branch pipelines, there is no way of configuring the gitlab connection.

[olhado]

All I am looking for is just supporting triggering the build. Additional items, like talking back to gitlab, would just be icing.

[coder-hugo]

@gtudan the commit status update already works for multi branch pipelines. You can set the GitLab connection by adding the following step to your Jenkinsfile

properties [[$class: 'GitLabConnectionProperty', gitLabConnection: '<your-gitlab-connection-name>']]

@olhado the multi branch pipeline support is the next point on "my list". So this should be implemented hopefully within the next weeks.

[marcore]

@coder-hugo That is the only setting needed ?
In a not multi branch job you have a flag in job configuration like this :
Build when a change is pushed to GitLab. GitLab CI Service URL: http://jenkins.url/project/JOB_NAME

so you can put that url in the gitlab webhooks section.

Using http://jenkins.url/project/MULTIBRANCH_PIPELINE_JOB_NAME in gitlab webhooks doesn't works. (Gitlab Checking CI status for sha-commit returns nothing)

As a workaround for commit status:

• I added a small node.js server that get invoked by gitlab webhooks and make a post on http://jenkins.url/job/MULTIBRANCH_PIPELINE_JOB_NAME/build with Jenkins authentication and force the multi branch pipeline job to reindex branches and so the job starts
• In the Jenkinsfile my steps are between try and catch and I call a function commitStatus with different values ("running", "success","failed") using directly Gitlab API. The function is:

def commitStatus(status) {
  sh "git rev-parse HEAD > .git/commit-id"
  def commit_id = readFile('.git/commit-id')
  def gitlabProjectId = "NUMERIC_ID_OF_GITLAB_PROJECT"
  def curlCommand = "curl -k --request POST "
  curlCommand += "--header 'cache-control: no-cache' "
  curlCommand += "--header 'content-type: application/x-www-form-urlencoded' "
  curlCommand += "--header 'private-token: ${env.GITLAB_PRIVATE_TOKEN}' "
  curlCommand += "--data 'state=${status}' "
  curlCommand += "--url https://gitlab.url/api/v3/projects/${gitlabProjectId}/statuses/${commit_id} "
  echo "${curlCommand}"
  sh "${curlCommand}"
}

As a note, I'm using Gitlab 8.9 , Jenkins 2.10 , Gitlab Plugin 1.2.5

[coder-hugo]

@marcore if you add the following step to your Jenkinsfile:

properties [[$class: 'GitLabConnectionProperty', gitLabConnection: '<your-gitlab-connection-name>']]

you can use the gitlabCommitStatus step of this plugin instead of your commitStatus function.

The only thing that is currently not implemented for multi branch pipeline jobs is the triggering of them.

[marcore]

@coder-hugo I must admit that when I add my own commitStatus function I was using a previous version of gitlab plugin (1.2.3 maybe) and the block gitlabCommitStatus { ... } completeley freeze my jenkins job.

I can confirm that with this plugin version (1.2.5) gitlabCommitStatus works as expected

[RomainVernoux]

In the meanwhile, the build can be triggered with the following workaround:

• Generate a Jenkins user token: go to <jenkins_url>/user/<jenkins_username/configure and click on Show API Token
• Set the job token: go to your job configuration, check Trigger build remotely (e.g., from scripts) and specify any token you like in the text field below.
• On the machine hosting GitLab, add a Git post-receive hook. For me, the hook folder was in /var/opt/gitlab/git-data/repositories/<project_owner>/<project_name>.git/custom_hooks/, so you want to add a file named post-receive there.
• configure the hook to call the Jenkins REST API. In the file mentionned above, put the following content: wget --auth-no-challenge --http-user=<jenkins_username> --http-password=<user_token> --secure-protocol=TLSv1 <jenkins_url>/job/<jenkins_job_name>/build?token=<job_token> &

This worked for me but I am also waiting for a cleaner way to trigger multibranch jobs from GitLab :)

[kilpatty]

@coder-hugo Could you clarify exactly what you did to fix this? I'm not super familiar with Jenkins code, so I couldn't garner it from your commits. I just installed 1.3.0 into Jenkins and my Multibranch pipelines are still not working as expected - just curious if it is this issue or another one.

[coder-hugo]

@Skilgarriff could you be a bit more concrete about what's not working as expected? Do you have configured the Webhooks in GitLab as described here? Furthermore have you read the configuration section for multibranch pipeline jobs in the README and adapted your Jenkinsfile accordingly?

[olhado]

I've just looked at the new docs for multi branch pipeline. I know we don't have access to the gitlab plugging env vars, but is the branch that is being built exposed as a variable in a different way?

The example checkout for multibranch implies you can't really modify any of the scm parameters.

Sent from my phone. Pardon the brevity and typos

On Jul 12, 2016, at 6:06 AM, Robin Müller [email protected] wrote:

@Skilgarriff could you be a bit more concrete about what's not working as expected? Do you have configured the Webhooks in GitLab as described here? Furthermore have you read the configuration section for multibranch pipeline jobs in the README and adapted your Jenkinsfile accordingly?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[schmitzhermes]

as far as I can see in my setup you can use env.BRANCH_NAME to determine the current branch.

[schmitzhermes]

I still have a problem with this one... I can trigger the build via Webhook and it actually starts the correct build. Moreover I can retrieve the current branch and the checkout scm surely checks this branch out.
That all is pretty nice... 👍

But at the point where gitlab-plugin tries to update my commit status, I am stuck. It fails with 500 Internal Server error on Gitlab's site.

Here is my Jenkinsfile:

properties properties: [[$class: 'GitLabConnectionProperty', gitLabConnection: 'Gitlab']]

node {
    stage "checkout"
    checkout scm

    println env.BRANCH_NAME // that works

    stage "build"
    gitlabCommitStatus("build") {
       println "aa" // this should work

    stage "test"
    gitlabCommitStatus("test") {
      sh 'a' // this should fail
    }
}

And here is the error that occurs in Gitlab production.log:

Started GET "/gitlab/api/v3/projects/some-group/some-project/repository/commits/cc2ddd3b63f6db200f0fd85b4380d648a1d7ea8c" for 127.0.0.1 at 2016-07-12 15:05:43 +0200
Processing by NamespacesController#show as JSON
  Parameters: {"id"=>"api/v3/projects/some-group/some-project/repository/commits/cc2ddd3b63f6db200f0fd85b4380d648a1d7ea8c"}
Completed 500 Internal Server Error in 81ms (ActiveRecord: 6.4ms)

ActionView::MissingTemplate (Missing template home/git/gitlab/public/404 with {:locale=>[:en], :formats=>[:json], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby, :coffee, :haml]}. Searched in:
  * "/home/git/gitlab/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/kaminari-0.16.3/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/doorkeeper-2.2.2/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/devise-3.5.4/app/views"
  * "/home/git/gitlab"
  * "/"
):
  app/controllers/application_controller.rb:144:in `render_404'
  app/controllers/namespaces_controller.rb:22:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

If I try the same request manually (via curl or Postman), it returns 404.
curl [...] http://myurl.com/gitlab/api/v3/projects/some-group%2Fsome-project/repository/commits/2b7d7dbob6d2c2a4aacf374c59537f341698ab111 returns 404.

If I, however, try to access the project via its numerical id (as Gitlab suggests in its docs), it returns the desired commit and 200 OK.

Can anybody tell me what the problem is? And can anybody tell me if its possible to use the Gitlab API with the group/project-notation? It never worked for me at all.

I'd be happy to provide more information.

[coder-hugo]

@schmitzhermes Within the GitLab log the %2F is missing as separator between namespace and project_name. So there is a system in between (e.g. an Apache reverse proxy) that replaces the %2F with a / which makes it impossible for GitLab to retrieve the correct project.
If there is an Apache reverse proxy in front of your GitLab please check if the AllowEncodedSlashes option is configured properly.

[schmitzhermes]

Oh, Jesus... that's good to know. I altered the url to not go through nginx proxy, but access it directly via docker network - works perfectly! Finally found the the reason why my manual curl also fails with 404 while using the group/project notation.

For the others coming from google: if you use nginx and cannot find a way around nginx: there is no such thing as AllowEncodedSlashes in nginx, but you have to rewrite the location (this is a link I found, but haven't tested: http://blog.jamesball.co.uk/2015/08/using-nginxapache-as-reverse-proxy-for.html)

[gtudan]

There was issue #337 a while back. The nginx documentation suggests that leaving the proxy-pass directive without an url should leave the URI as is, but that's basically what the default gitlab-config does and it still decodes the %2F. I also tried adding the $request_uri Param to get the URL before any rewrites, but that didn't work either.

I still wonder why there aren't more users bumping into this issue, as nginx as reverse proxy is the default setup of gitlab.

[coder-hugo]

The nginx config of the dockerized GitLab (see README) is working fine. Maybe you can have a look at this and compare it with yours.

[gtudan]

Mhhh... that's pretty much the same one the from-source-install and omnibus is using, so that doesn't seem to fix the problem.

[coder-hugo]

Can you have a look to the access log of the nginx and check if the URL contains the %2F as separator between namespace and project_name. Just to ensure the nginx is the system that replaces the %2F by a /.

[gtudan]

In m access log I have

"GET /api/v3/projects/VV%2Fcustomer-frontend/repository/commits/1cf168974538ee5c65e73cd9a4d31393eb4518d1 HTTP/1.1" 500 48 "-" "-"

In the gitlab-workhorse log, the escaping is still there:

gitlab.dev.tech.visualvest.de @ - - [2016-07-13 10:18:45.189223163 +0200 CEST] "GET /api/v3/projects/VV%2Fcustomer-frontend/repository/commits/1cf168974538ee5c65e73cd9a4d31393eb4518d1 HTTP/1.1" 500 48 "" "" 0.166421

And then, there is the production.log

Started GET "/api/v3/projects/VV/customer-frontend/repository/commits/1cf168974538ee5c65e73cd9a4d31393eb4518d1" for 127.0.0.1 at 2016-07-13 10:18:45 +0200
Processing by NamespacesController#show as JSON
  Parameters: {"id"=>"api/v3/projects/VV/customer-frontend/repository/commits/1cf168974538ee5c65e73cd9a4d31393eb4518d1"}
Completed 500 Internal Server Error in 136ms (ActiveRecord: 15.6ms)

ActionView::MissingTemplate (Missing template home/git/gitlab/public/404 with {:locale=>[:en], :formats=>[:json], :variants=>[], :handlers=>[:erb, :builder, :raw, :ruby, :coffee, :haml]}. Searched in:
  * "/home/git/gitlab/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/kaminari-0.17.0/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/doorkeeper-3.1.0/app/views"
  * "/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/devise-4.1.1/app/views"
  * "/home/git/gitlab"
  * "/"
):
  app/controllers/application_controller.rb:142:in `render_404'
  app/controllers/namespaces_controller.rb:22:in `show'
  lib/gitlab/middleware/go.rb:16:in `call'

[coder-hugo]

Ok, so if the config is the same I'd assume the version of the nginx is different. The version of the nginx of the dockerized GitLab is 1.10.1. What is yours?

[gtudan]

I finally got things working - as the encoding survives nginx, I started looking at the gitlab workhorse. There are several issues around URL-Encoding, but all of them have been fixed in 0.7 and later. But I noticed that the go-Version I was using to compile the workhorse was older than recommended (1.4.x), so I updated to 1.6. and recompiled. And finally, things suddenly started working 🎉

So it's like @coder-hugo suggested: the plugin is fine and neither should the nginx-config with the default config cause any trouble. If you still bump into this issue, take a look at the workhorse version and your Go-Compiler.

[slmjy]

Does someone have a workaround for building merge requests, like in pipeline job with pipeline multibranch?

[schmitzhermes]

I got one last problem with multibranch-triggers (at least one last major-problem) ... [ci-skip] obviously does not work, because there is no chance to configure the plugin.

Do you guys have any ideas? Can we reuse the ci-skip logic in mutlibranch-builds (it'd be enabled by default then)?

[schmitzhermes]

@slmjy why workaround? the push/MR action triggers the multibranch build and the multibranch-build notifies Gitlab about the commit status via gitlabCommitStatus - et voila, works.

obviously there is - right now - no chance to comment the MR via the plugin or something like that, because there is no chance to configure the plugin in a multibranch build.

[coder-hugo]

@schmitzhermes IMHO it's no good idea to prevent the CI from building stuff by adding a special string to the commit message. If you don't want the CI building your stuff you should do this by setting up a branch filter in the git configuration of the multibranch pipeline instead of wasting the history with [ci-skip] messages.
As this feature was already implemented for FreeStyleJobs and "normal" PipelineJobs when I became the lead maintainer I haven't removed it don't break the workflow of those who are using this feature. But by switching to the multibranch pipelines your are changing your workflow anyway so I don't see any good reason to add this feature for multibranch pipelines.

[schmitzhermes]

Hmm @coder-hugo I do see one (or two) good situation(s).
First, and most important: what about locking dependencies? To make my gradle build immutable I use a Netflix plugin that locks the version (i.e. concrete versions instead of 1.+) of my dependencies. The plugin does that in generating a JSON file that has to be checked in into the repository. With this technique you know that the build will produce the same result if you switch back to a specific git tag. This dependency-locking file is obviously committed (and pushed) by Jenkins.

Another reason would be: versioning of npm-modules. While you can use git tags for this purpose, npm relies on the package.json file, that holds the current version of that asset. Obviously that file has to be committed to the git repository.

[slmjy]

@schmitzhermes I meant when building merge requests - merging source branch into target first and then running the build. That's what not possible with multibranch obviously, so I'm asking about possible workarounds. One that comes in mind is having a freestylejob, that will run particular pipeline branch with parameters for merging, that can be somehow supported in pipeline script

[coder-hugo]

@schmitzhermes if you want to prevent the Jenkins from building some special commits you have to do this within the Jenkinsfile. Which is IMHO the better approach as everything else won't ensure that these commits aren't built.
Furthermore it's impossible for the plugin to don't trigger builds for [ci-skip] commits. As the plugin just notifies the multibranch pipeline job about changes and the job itself triggers the builds for the changed branches. Just an example what would happen if the plugin would have the ci-skip implemented:

1. You make a commit and push it
2. The plugin receives the event from GitLab and notifies the mutlibranch pipeline job about
3. The multibranch pipeline jobs starts a branch indexing (fetches all remote branches and looks for changes)
4. The branch indexing starts a build for the changed branch
5. The Jenkins builds this commit, creates the special commit and pushes the changes
6. The plugin receives the event from GitLab and doesn't notify the mutlibranch pipeline job about because of the ci-skip
7. You add a commit on a different brach and push it
8. The plugin receives the event from GitLab and notifies the mutlibranch pipeline job about
9. The multibranch pipeline jobs starts a branch indexing
10. The branch indexing starts a build for all changed branches (including the branch with the ci-skip)

[omehegan]

I'm confused... I've updated to 1.4.0 of the plugin and then created a test Pipeline Multibranch job. The job builds OK manually, but I don't see an option to trigger it via GitLab. How am I supposed to do that?

[coder-hugo]

Just configure the WebHook in GitLab as usual and the build will be triggered automatically. For mutlibranch pipeline jobs you can't configure triggers.

[omehegan]

Hmm, that is not intuitive... so it looks like we still create a hook to http://gitlab/project/BUILD_NAME? It's too bad we can't surface that in the multibranch project UI. That hook seems to work, should I update the README accordingly?

[minid33]

This ticket is closed but will there be a better workflow than adding some web hooks to get multibranch pipelines working preferably with the goal of supplying parameters or env vars from gitlab?

[omehegan]

@minid33 this is a good question. From what I have been able to tell, multibranch jobs are simply not intended to be parameterized. I think we would have to establish what the use case is for being able to pass parameters in, before we could advocate for that support.

My impression is that Multibranch jobs which set additional parameters, such as CHANGE_AUTHOR and CHANGE_AUTHOR_EMAIL and so on, do so by querying a relevant API after the build is triggered. For example, hitting the GitHub API. See e.g. https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestAction.java#L79

I think the right way forward is to extend SCMSource, rather than GitSCMSource as we seem to be doing right now. And then implement ChangeRequestAction to get access to some additional stuff. I think there is more insight on this here: https://wiki.jenkins-ci.org/display/JENKINS/Extension+points#Extensionpoints-jenkins.scm.api.SCMSource

Unfortunately I'm not up to the task of changing stuff like this... I have to leave it to @coder-hugo or others :)

[dylan-underwood]

@coder-hugo and @omehegan I've read through this issue a couple times and am still a little confused on the available functionality.

Can a Merge Request event from a webhook in GitlLab trigger a Multibranch Pipeline job in Jenkins?

I am able to trigger my Multibranch Pipeline job from a Push event, but a Merge Request event doesn't seem to trigger the job. Is this functionality simply not available? Am I doing something wrong? Is there any other workaround? Please let me know when you can. Thank you.

My Jenkins version is 2.19.1
My GitLab Plugin is version 1.4.2
My GitLab is Community Edition 8.9.3

[omehegan]

@dylan-underwood no. Per the note here: https://github.com/jenkinsci/gitlab-plugin#git-configuration-for-multibranch-pipelineworkflow-jobs because multibranch jobs can't receive parameters from the web hook, merge request triggers don't really make sense. For example we can't pass in the source and target branches, so the job can't test the merge before build, etc. So the plugin only listens for push webhooks when the job is a multibranch type.

[carlomorelli]

Good morning everyone, I know that the issue is closed, just commenting to ask if there are any news in the gitlab-plugin supporting the Merge Request env variables in the Multibranch pipelines.

Until we don't have access to variables ${env.gitlabSourceBranch} and ${env.gitlabTargetBranch}, it seems that we Gitlab users are stuck with the traditional pipeline if we want to do the infamous "merge target branch before build" action... right?

However, I understand that the Github and Bitbucket plugins already fully support Pullrequests actions... :(

[omehegan]

@trance1st maybe you want to look at this and reopen if necessary. It seems like there should be a way to do this, but it's a question of what the right way to support it in Jenkins would be.

@carlomorelli my understanding is that those plugins integrate more tightly with the GitHub and Bitbucket APIs than our plugin currently does with GitLab.

[philippe-granet]

These plugins (under dev, not prod ready) can build Gitlab Merge Request in Multibranch pipelines:
https://github.com/Argelbargel/gitlab-branch-source-plugin
https://github.com/kohsuke/gitlab-branch-source-plugin

[Hermain]

@philippe-granet Can you access the gitlab environment variables which come from the webhook using these plugins?
This has been bugging me for a year now and I guess it won't ever be fixed. That's a real shame

[matthiasstock]

FYI I came across a Google Summer of Code Project Multibranch Pipeline for GitLab that could improve the situation.