Gitlab-plugin: Merge request hooks are ignored for multibranch jobs

[sturmf]

Issue

Even though GitLab calls the correct webhook and the Jenkins logger shows a sensible output for a MergeRequest, Jenkins does not schedule one. In the documentation of the gitlab-plugin there is information about:

• In the Advanced settings, set its Name to origin and its refspec to +refs/heads/_:refs/remotes/origin/_ +refs/merge-requests/_/head:refs/remotes/origin/merge-requests/_

I can't find those fields for a multibranch pipeline project :-(

What else could I be missing to get this working? Regular pushes to GitLab build and now I also get the build status pushed back to GitLab. The GitLab api key also can be tested successfully and the jenkins user has developer access to the repository where the merge request came from.

Context

• Gitlab plugin version: 141
• Gitlab version: 8.11.5
• Jenkins version: 2.7.4
  

  Logs & Traces

Problem description

Here the gitlab-plugin log (sensitive information replaced)

Sep 13, 2016 5:52:38 PM INFO com.dabsquared.gitlabjenkins.webhook.GitLabWebHook getDynamic
WebHook called with url: /project/myproject
Sep 13, 2016 5:52:38 PM FINE com.dabsquared.gitlabjenkins.webhook.build.MergeRequestBuildAction
MergeRequest: {
  "object_kind" : "merge_request",
  "user" : {
    "name" : "STURM",
    "username" : "STURM",
    "avatar_url" : "http://www.gravatar.com/avatar/bdd2eb83653c8aaffd7fd10671083dd1?s=80&d=identicon"
  },
  "project" : {
    "name" : "myproject",
    "description" : "The  myproject is a... ",
    "web_url" : "http://HOSTNAME/groupname/myproject",
    "avatar_url" : null,
    "git_ssh_url" : "git@HOSTNAME:groupname/myproject.git",
    "git_http_url" : "http://HOSTNAME/groupname/myproject.git",
    "namespace" : "groupname",
    "visibility_level" : 10,
    "path_with_namespace" : "groupname/myproject",
    "default_branch" : "master",
    "homepage" : "http://HOSTNAME/groupname/myproject",
    "url" : "git@HOSTNAME:groupname/myproject.git",
    "ssh_url" : "git@HOSTNAME:groupname/myproject.git",
    "http_url" : "http://HOSTNAME/groupname/myproject.git"
  },
  "object_attributes" : {
    "id" : 27,
    "target_branch" : "master",
    "source_branch" : "jenkinstest2",
    "source_project_id" : 5,
    "author_id" : 6,
    "assignee_id" : 6,
    "title" : "one more jenkins try",
    "created_at" : "2016-09-13 15:52:38 UTC",
    "updated_at" : "2016-09-13 15:52:38 UTC",
    "milestone_id" : null,
    "state" : "opened",
    "merge_status" : "unchecked",
    "target_project_id" : 1,
    "iid" : 27,
    "description" : "",
    "position" : 0,
    "locked_at" : null,
    "updated_by_id" : null,
    "merge_error" : null,
    "merge_params" : {
      "force_remove_source_branch" : null
    },
    "merge_when_build_succeeds" : false,
    "merge_user_id" : null,
    "merge_commit_sha" : null,
    "deleted_at" : null,
    "in_progress_merge_commit_sha" : null,
    "source" : {
      "name" : "myproject",
      "description" : "The  myproject is a... ",
      "web_url" : "http://HOSTNAME/STURM/myproject",
      "avatar_url" : null,
      "git_ssh_url" : "git@HOSTNAME:STURM/myproject.git",
      "git_http_url" : "http://HOSTNAME/STURM/myproject.git",
      "namespace" : "STURM",
      "visibility_level" : 10,
      "path_with_namespace" : "STURM/myproject",
      "default_branch" : "master",
      "homepage" : "http://HOSTNAME/STURM/myproject",
      "url" : "git@HOSTNAME:STURM/myproject.git",
      "ssh_url" : "git@HOSTNAME:STURM/myproject.git",
      "http_url" : "http://HOSTNAME/STURM/myproject.git"
    },
    "target" : {
      "name" : "myproject",
      "description" : "The  myproject is a... ",
      "web_url" : "http://HOSTNAME/groupname/myproject",
      "avatar_url" : null,
      "git_ssh_url" : "git@HOSTNAME:groupname/myproject.git",
      "git_http_url" : "http://HOSTNAME/groupname/myproject.git",
      "namespace" : "groupname",
      "visibility_level" : 10,
      "path_with_namespace" : "groupname/myproject",
      "default_branch" : "master",
      "homepage" : "http://HOSTNAME/groupname/myproject",
      "url" : "git@HOSTNAME:groupname/myproject.git",
      "ssh_url" : "git@HOSTNAME:groupname/myproject.git",
      "http_url" : "http://HOSTNAME/groupname/myproject.git"
    },
    "last_commit" : {
      "id" : "3209eecdf6be5601ec1e248b7c2fc9c3712defce",
      "message" : "one more jenkins try\n",
      "timestamp" : "2016-09-13T17:50:45+02:00",
      "url" : "http://HOSTNAME/groupname/myproject/commit/3209eecdf6be5601ec1e248b7c2fc9c3712defce",
      "author" : {
        "name" : "Fabian Sturm",
        "email" : "[email protected]"
      }
    },
    "work_in_progress" : false,
    "url" : "http://HOSTNAME/groupname/myproject/merge_requests/27",
    "action" : "open"
  },
  "repository" : {
    "name" : "myproject",
    "url" : "git@HOSTNAME:groupname/myproject.git",
    "description" : "The  myproject is a... ",
    "homepage" : "http://HOSTNAME/groupname/myproject"
  },
  "assignee" : {
    "name" : "STURM",
    "username" : "STURM",
    "avatar_url" : "http://www.gravatar.com/avatar/bdd2eb83653c8aaffd7fd10671083dd1?s=80&d=identicon"
  }
}

[coder-hugo]

This is currently not supported. Please have a look to the README for further information.

[sturmf]

Thanks for the info. I couldn't deduce this information from the README, at least I don't see a place where it says that this is not possible.

I guess my hope it would be supported blinded me :-)

Would be great to add this in some further version!

[coder-hugo]

In my last comment there is a link directly to this part of the README

Note: none of the GitLab environment variables are available for mulitbranch pipeline jobs as there is no way to pass some additional data to a multibranch pipeline build while notifying a multibranch pipeline job about SCM changes. Due to this the plugin just listens for GitLab Push Hooks for multibranch pipeline jobs.

[sturmf]

Hmm I guess the point of this sentence is lost on me as a non native english speaker. Sorry for that!

I thought this was just some technical detail but I did not get the implication since the plugin clearly recieves all necessary information about the merge request as can be seen in the log. So I would suggest to add ".. and due to that we can no automatically build merge requests in a multibranch pipelne build".

[cjbush]

Does this also apply to addGitLabMRComment? I don't seem to be getting comments on my merge requests using a multibranch pipeline.

[omehegan]

@cjbush that sounds like a separate issue. Please open a new one if you're still experiencing that.

[philippe-granet]

These plugins (under dev, not prod ready) can build Gitlab Merge Request in Multibranch pipelines:
https://github.com/Argelbargel/gitlab-branch-source-plugin
https://github.com/kohsuke/gitlab-branch-source-plugin

[amarruedo]

I also did not quite understand from the README that merge-request job triggering is not supported for multibranch pipeline jobs. I can't imagine why it can't trigger a folder scan so the newly created merge-request is scheduled and build on jenkins (I guess it would be the same functionality as when a push event is sent).

So, to be able to schedule merge-requests, I guess the only chance that we have toady is to totally bypass the plugin and configure gitlab webhook for merge-request event using http://JENKINS_URL/job/PROJECT_NAME/build in the webhook url. Am I right?

[omehegan]

@amarruedo as you said, triggering on an MR event from GitLab would be equivalent to triggering on a push event, in terms of what the plugin will do - it only performs branch indexing. Is there some reason why just configuring the push webhook wouldn't work for you?

[amarruedo]

@omehegan somebody works on a branch, and when the work is finished, that person creates a merge-request. We would like that merge-request triggering a build, and if it succeeds, we would like it automatically merged. If you only configure the push webhook, nothing is scheduled on Jenkins, unless you push something to that merge-request. So apparently when you only configure for pushes, the only way to schedule a build on a merge-request is to push something on that merge-request even if you already have finished working on the code of the branch you want to merge.

[omehegan]

@amarruedo that makes sense. I don't really know why @coder-hugo wrote it this way, but he is no longer working on the plugin. I will re-open this issue and mark it as a bug, but I don't know if I will be able to fix it.

[amarruedo]

@omehegan I've been looking to the code and I think the fix should be easy to implement. Only thing to do (I guess...) is to do the same thing on push build action (the tag build action is the same as push build action as seen here) and on merge-request build action .

We need on Execute function from MergeRequestBuildAction this code and associated SCMSourceOwnerNotifier class.

Inside this class there is a call to onSCMSourceUpdated, which I think is the function that trigers folder scan.

When I geet some spare time I'll try to check by myself if these changes work.

[omehegan]

@amarruedo a colleague also looked at this issue, but wasn't sure that triggering branch indexing from webhooks was a good practice.

This would solve the issue but is bad IMO. Branch Indexing is designed to scan for branches / MR only and should be triggered once a day for active repository. Branch indexing is processed on the master. Triggering branch indexing aggressively could have negative impact on master's performances.

The GitLab plugin could benefits from an SCMSource implementation so that GitLab repository are retrieved via REST API and Push / Merge request events are processed by the Events API.

This implies that we would need to complete #499...

[ondrejtomcik]

Please, is there a chance that this will be done? I don't understand why MR event from the GitLab cannot trigger build in a same way as push event.

[omehegan]

@ondrejtomcik it could be done, but as noted above we are growing concerned that the current implementation of Multibranch support is not optimal from a performance perspective. If @amarruedo wants to work on his proposed solution we can consider it though. I desperately want someone to work on #499.

[amarruedo]

The workaround I proposed should take 2 minutes of development (just copy/paste the bits of code I suggested), and as @omehegan says, it is far from ideal.

On our side, due to this issue, and also to this one (not related with this pulgin), we are planning to move our CI system to Gitlabs own CI environment.

[ondrejtomcik]

The workaround I proposed should take 2 minutes of development (just copy/paste the bits of code I suggested), and as @omehegan says, it is far from ideal.

On our side, due to this issue, and also to this one (not related with this pulgin), we are planning to move our CI system to Gitlabs own CI environment.

We are considering it too.

[omehegan]

I've made the changes suggested by @amarruedo and pushed a snapshot build here. I haven't had a chance to perform manual testing yet, I'll try to get to that next week. But I would welcome anyone else's feedback in the meantime.

[Leanivets]

@omehegan after installation snapshot plugin version http://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/gitlab-plugin/1.5.12-SNAPSHOT/gitlab-plugin-1.5.12-20181123.055442-1.hpi "Merge Request Hook" on GitLab returned 200 but Jenkins multibranch jod not trigger yet

[ondrejtomcik]

FYI: https://wiki.jenkins.io/display/JENKINS/Generic+Webhook+Trigger+Plugin

Just testing it, we have metadata and works with MR.

[omehegan]

@amarruedo any thoughts as to why the change you proposed doesn't seem to have worked? See #857.

[matejdro]

I'm not sure this auto-merge strategy would work:

https://github.com/jenkinsci/gitlab-plugin/blob/b89cb7ff3696642fa546f7dbba7494e61251f644/src/main/java/com/dabsquared/gitlabjenkins/workflow/AcceptGitLabMergeRequestStep.java#L88

Merge step requires cause to be gitlab cause. And you said that no gitlab data can be passed to multibranch build.

[alinpa]

Hello,

I have the same issue.
Do you have any updates on this issue ?

[derrickgw]

I have been able to (mostly) work around this issue using the following function. This function will let me know if there is an active merge request for the current commit, and I can then run the longer tests. It isn't perfect because if you create a merge request after pushing everything, it doesn't re-trigger, but if you manually run the job or push again, then it will detect the MR.

Ideally this sort of simple check would be made available by the plugin so I don't have to commit my api token. It could even return an object that had more information, like the target branch.

```java
def is_open_mr(project_name, branch_name) {
project_id = project_name.replace('/', '%2f')
url = 'https://gitlabee/api/v4/'
url += "projects/$project_id/merge_requests"
url += "?source_branch=$branch_name"
url += "&wip=no"
url += "&state=opened"
def token = 'token'

response = httpRequest url: url,
    customHeaders: [[maskValue: true, name: 'Private-Token', value: token]]
def j = new groovy.json.JsonSlurper().parseText(response.content)
return (j.size() > 0)

}```

[bhavikbhavsar]

Any update on this issue?
I tried to use the job type as pipeline instead of multibranch pipeline but it is not able to detect shared library.

[subir0071]

Multibranch scan webhook worked for me --> https://plugins.jenkins.io/multibranch-scan-webhook-trigger/

[juliofalbo]

Any news?

[omehegan]

For multibranch projects using GitLab, going forward I would recommend that users adopt the GitLab Branch Source plugin. Its design is focused solely on support for multibranch projects, and handles that in a much more scalable and performant way than this plugin does.