Gitlab-plugin: Multibranch Pipeline not updating Gitlab Status

[shikage]

Issue

Context

• Gitlab plugin version: 1.5.2
• Gitlab version: 10.2.4 CE
• Jenkins version: 2.89.2
• Job type: Multibranch Pipeline

Logs & Traces

Jenkins Log for GitLab Plugin

Dec 17, 2017 7:16:21 PM INFO com.dabsquared.gitlabjenkins.webhook.GitLabWebHook getDynamic
WebHook called with url: /jenkins/project/spacegen-lib
Dec 17, 2017 7:16:21 PM FINE com.dabsquared.gitlabjenkins.webhook.build.PushBuildAction
Push: {
  [...]
  }
}
Dec 17, 2017 7:16:21 PM FINE com.dabsquared.gitlabjenkins.webhook.build.PushBuildAction
Notify scmSourceOwner spacegen-lib about changes for git@localhost:Spacesim/spacegen-lib.git

Jenkins File:

properties([[$class: 'GitLabConnectionProperty', gitLabConnection: 'Orkwerks Gitlab']])

node {
    stage "checkout"
    checkout scm

    stage "build"
    gitlabCommitStatus("build") {
        sh 'mvn clean install -DskipTests'
    }

    stage "test"
    gitlabCommitStatus("test") {
        sh 'mvn test'
    }
}

Problem description

Gitlab is not being updated with the build status after Jenkins completes the build. The log messages seem to indicate that the update is supposed to be getting triggered but nothing shows up in activity, branches or commits to indicate this. I am not seeing much in the GitLab log to show anything going back to the server either.

[Azarko]

Hi.

I had a problem similar to yours.
In my case problem was that user which I use for gitlab connection (with Personal Access Token ) haven't write permission to gitlab project (require 'Developer' permission at least).

Anyway, you should watch GitLab response code and body. See the section User support in the project's readme to set up logging in jenkins. In my case there was

Status: 403 Forbidden 
failed to print message Failed to update Gitlab commit status for project '***': HTTP 403 Forbidden. 

and when I set up permissions:

Status: 201 Created

[shikage]

Unfortunately, I don't see any messages of that sort in the log I setup in Jenkins. I get a JSON output of the details of the GitLab project and then the following notify message:
Dec 22, 2017 12:35:17 PM FINE com.dabsquared.gitlabjenkins.webhook.build.PushBuildAction
Notify scmSourceOwner about changes for git@localhost:/.git

My jenkins and gitlab are on the same system and so to avoid firewall issues I point the ssh to localhost. But that is it. No indication of the responses from GitLab in the Jenkins log.

[shikage]

I should add to this that I have not tried with a Freestyle project and that does successfully post the status back to GitLab.

[omehegan]

@shikage you did test with a Freestyle job, and that works?

[shikage]

@omehegan I did test with a freestyle job.. I see now my last post was mistyped. I HAVE tried this, it did work as expected. It is just multibranch pipeline that is encountering problems.

[omehegan]

@shikage your properties configuration does not match the documentation. Can you try:
properties([gitLabConnection('Orkwerks Gitlab')])

[shikage]

I have tried with the properties you suggested as my first line. It has the same effect. I am fairly certain that is how I started but I found a number of stackoverflow and similar posts suggesting the other property setting had solved problems for them. To be sure I've just tried again with both and it still does not update the status.

[mabrowning]

I'm in the same boat.

A standard "Pipeline" jenkins job adds GitLab build status
The same setup as a multibranch pipeline does not add GitLab build status.

[omehegan]

@mabrowning can you enable the additional logging described in the User Support section of the doc? Then please attach logs from a working Pipeline job and non-working Multibranch job.

[mabrowning]

Sure thing. I did have them enabled and was looking at them, but I had both hooks/builds enabled simultaneously and couldn't sort out the logstream independently. I've just done a test to separate them. I force pushed an amended commit, so the commit ids are different, but everything else is the same (other than which webhook/Jenkins project). Both jenkins projects are set to pull Jenkinsfile from SCM and build correctly/successfully other than the gitlab updates.

EDIT. After updating to Version 1.5.1 of the plugin, something interesting appears in the Multibranch pipeline job log

(Updated) Multibranch Pipeline log:

Mar 23, 2018 10:44:30 AM FINE com.dabsquared.gitlabjenkins.webhook.build.PushBuildAction
Notify scmSourceOwner PROJECT_NAME about changes for [email protected]:GROUP_NAME/PROJECT_NAME
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
Retrieving gitlab project ids
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Retrieving the gitlab project id from remote url [email protected]:GROUP_NAME/PROJECT_NAME
Mar 23, 2018 10:45:01 AM WARNING com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Did not match project id in remote url.
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
Retrieving gitlab project ids
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Retrieving the gitlab project id from remote url [email protected]:GROUP_NAME/PROJECT_NAME
Mar 23, 2018 10:45:01 AM WARNING com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Did not match project id in remote url.
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
Retrieving gitlab project ids
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Retrieving the gitlab project id from remote url [email protected]:GROUP_NAME/PROJECT_NAME
Mar 23, 2018 10:45:01 AM WARNING com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Did not match project id in remote url.
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
Retrieving gitlab project ids
Mar 23, 2018 10:45:01 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Retrieving the gitlab project id from remote url [email protected]:GROUP_NAME/PROJECT_NAME
Mar 23, 2018 10:45:01 AM WARNING com.dabsquared.gitlabjenkins.util.CommitStatusUpdater addGitLabBranchBuild
Did not match project id in remote url.

"Normal" Pipeline job (note that I cancelled the build during one stage, but that successfully made it to gitlab):

Mar 23, 2018 9:19:19 AM INFO com.dabsquared.gitlabjenkins.webhook.GitLabWebHook getDynamic
WebHook called with url: /project/group/gitlab-test
Mar 23, 2018 9:19:19 AM FINE com.dabsquared.gitlabjenkins.webhook.build.PushBuildAction
Push: {
  "object_kind" : "push",
  "event_name" : "push",
  "before" : "6d08dde5ebec1ee8391c2776d4f5a4d03c8efd41",
  "after" : "00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
  "ref" : "refs/heads/more-jenkins",
  "checkout_sha" : "00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
  "message" : null,
  "user_id" : 10,
  "user_name" : "Mark Browning",
  "user_username" : "mbrowning",
  "user_email" : "[email protected]",
  "user_avatar" : "https://secure.gravatar.com/",
  "project_id" : 178,
  "project" : {
    "name" : "project",
    "description" : "Project Description"",
    "web_url" : "https://git.company.com/group/project",
    "avatar_url" : null,
    "git_ssh_url" : "[email protected]:group/project.git",
    "git_http_url" : "https://git.company.com/group/project.git",
    "namespace" : "group",
    "visibility_level" : 10,
    "path_with_namespace" : "group/project",
    "default_branch" : "master",
    "ci_config_path" : null,
    "homepage" : "https://git.company.com/group/project",
    "url" : "[email protected]:group/project.git",
    "ssh_url" : "[email protected]:group/project.git",
    "http_url" : "https://git.company.com/group/project.git"
  },
  "commits" : [ {
    "id" : "00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
    "message" : "Update comments\n",
    "timestamp" : "2018-03-23T09:15:45-05:00",
    "url" : "https://git.company.com/group/project/commit/00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
    "author" : {
      "name" : "Mark Browning",
      "email" : "[email protected]"
    },
    "added" : [ ],
    "modified" : [ "Jenkinsfile" ],
    "removed" : [ ]
  } ],
  "total_commits_count" : 1,
  "repository" : {
    "name" : "project",
    "url" : "[email protected]:group/project.git",
    "description" : "Project Description"",
    "homepage" : "https://git.company.com/group/project",
    "git_http_url" : "https://git.company.com/group/project.git",
    "git_ssh_url" : "[email protected]:group/project.git",
    "visibility_level" : 10
  }
}
Mar 23, 2018 9:19:19 AM INFO com.dabsquared.gitlabjenkins.trigger.handler.AbstractWebHookTriggerHandler handle
group/gitlab-test triggered for push.
Mar 23, 2018 9:19:38 AM INFO com.dabsquared.gitlabjenkins.util.CommitStatusUpdater retrieveGitlabProjectIds
Retrieving gitlab project ids
Mar 23, 2018 9:19:38 AM FINEST com.dabsquared.gitlabjenkins.gitlab.GitLabClientBuilder
Call GitLab:
HTTP method: GET
URL: https://git.company.com/api/v3/projects/178/repository/commits/00b7c954b6dd94e1b1eb731880daa1df8d1c2721
Request headers: [
Accept = [application/json],
PRIVATE-TOKEN = [****FILTERED****]
]
Mar 23, 2018 9:19:38 AM FINEST com.dabsquared.gitlabjenkins.gitlab.GitLabClientBuilder
Got response from GitLab:
URL: https://git.company.com/api/v3/projects/178/repository/commits/00b7c954b6dd94e1b1eb731880daa1df8d1c2721
Status: 200 OK
Response headers: [
Cache-Control = [max-age=0, private, must-revalidate],
Connection = [keep-alive],
Content-Length = [537],
Content-Type = [application/json],
Date = [Fri, 23 Mar 2018 14:17:40 GMT],
Etag = [W/"46fc9469c1eed2ad106972a2c2bc5265"],
Server = [nginx],
Strict-Transport-Security = [max-age=31536000],
Vary = [Origin],
X-Frame-Options = [SAMEORIGIN],
X-Request-Id = [db0ab016-8bed-4fee-ad8a-4e1d0d14c975],
X-Runtime = [0.071676]
]
Response body: {
  "id" : "00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
  "short_id" : "00b7c954",
  "title" : "Update comments",
  "created_at" : "2018-03-23T09:15:45.000-05:00",
  "parent_ids" : [ "cc3e20ba20792c545057e95b0065389f2964172d" ],
  "message" : "Update comments\n",
  "author_name" : "Mark Browning",
  "author_email" : "[email protected]",
  "authored_date" : "2018-03-23T09:09:51.000-05:00",
  "committer_name" : "Mark Browning",
  "committer_email" : "[email protected]",
  "committed_date" : "2018-03-23T09:15:45.000-05:00",
  "stats" : {
    "additions" : 1,
    "deletions" : 1,
    "total" : 2
  },
  "status" : "failed"
}
Mar 23, 2018 9:19:38 AM FINEST com.dabsquared.gitlabjenkins.gitlab.GitLabClientBuilder
Call GitLab:
HTTP method: POST
URL: https://git.company.com/api/v3/projects/178/statuses/00b7c954b6dd94e1b1eb731880daa1df8d1c2721
Request headers: [
Accept = [application/json],
Content-Type = [application/x-www-form-urlencoded],
PRIVATE-TOKEN = [****FILTERED****]
]
Mar 23, 2018 9:19:38 AM FINEST com.dabsquared.gitlabjenkins.gitlab.GitLabClientBuilder
Got response from GitLab:
URL: https://git.company.com/api/v3/projects/178/statuses/00b7c954b6dd94e1b1eb731880daa1df8d1c2721
Status: 201 Created
Response headers: [
Cache-Control = [max-age=0, private, must-revalidate],
Connection = [keep-alive],
Content-Length = [529],
Content-Type = [application/json],
Date = [Fri, 23 Mar 2018 14:17:40 GMT],
Etag = [W/"420a8c846c7ffa091a6cf6aec7571812"],
Server = [nginx],
Strict-Transport-Security = [max-age=31536000],
Vary = [Origin],
X-Frame-Options = [SAMEORIGIN],
X-Request-Id = [babb6541-5167-4923-8c2b-571941b55c2e],
X-Runtime = [0.061108]
]
Response body: {
  "id" : 65,
  "sha" : "00b7c954b6dd94e1b1eb731880daa1df8d1c2721",
  "ref" : "more-jenkins",
  "status" : "pending",
  "name" : "build-Linux",
  "target_url" : "https://jenkins.company.com/job/group/job/gitlab-test/6/",
  "description" : null,
  "created_at" : "2018-03-23T14:17:40.095Z",
  "started_at" : null,
  "finished_at" : null,
  "allow_failure" : false,
  "coverage" : null,
  "author" : {
    "name" : "Jenkins CI",
    "username" : "jenkins",
    "id" : 90,
    "state" : "active",
    "avatar_url" : "https://git.company.com/uploads/-/system/user/",
    "web_url" : "https://git.company.com/jenkins"
  }
}
...

[mabrowning]

Digging through the code in question, it seems that the multibranch pipeline job doesn't have the correct GitlabClient set in it, as the changeBuildStatus method on the interface is called (proved to myself by induction by which logs AREN'T called). However, none of the API/proxies ever get called.

This is plausible as the normal pipeline job explicitly adds a Gitlab connection to the build properties, but the Multibranch has to pull it from the Jenkinsfile ( properties([ gitLabConnection('my-correct-gitlab-connection-id')] ) ).

[mabrowning]

Hmm..I was inadvertently on an old version of this plugin (1.4.8). I'm going to update to 1.5.1 since there were some major revisions to the GitlabClient interface.

[mabrowning]

Success! After changing our git URL from ssh to https (https://git.company.com/group/project.git, with a user/password credential of (literally)gitlab-ci-token/JENKINS-IMPERSONATION-TOKEN), multibranch pipeline build correctly updates the gitlab build status.

I did need to fix our WIndows nodes; builds were hanging on them during git fetch because the Git Credential Manager for windows was installed.

[omehegan]

@mabrowning hmmm. I don't really understand why this fixed the issue for you. You should be able to use either SSH or HTTP and still have the commit status update work... Do you have any theory about why it would matter?

[mabrowning]

As I was digging into the code to explain why it wasn't matching ssh vs https, I discovered the real reason: my ssh url did not include the trailing .git. Down in ProjectIdUtil, the regex ( ^/?(?<projectId>.*)(\\.git)$ ) currently _requires_ it, throwing the "Did not match project id in remote url." error if it isn't present. An alternative regex ( ^/?(?<projectId>.*?)(\\.git)?$ ) handles with or without the .git suffix.

Notably, the git url that @shikage seems to be using (git@localhost:/.git) also is "malformed" in this regex (doesn't throw, but returns the empty string). I don't know what it should return, as it seems to be missing that info, but a better error message would probably be good.

This was also reported in #462.

[hervedevos]

I have also an issue related to multibranch commit status updates.

In my case, here what's happening:

jenkins log :
INFO: Retrieving the gitlab project id from remote url https:///gitlab/test/test-project.git
gitlab log:
Started GET "/gitlab/api/v4/projects/test/test-project/repository/commits/b133f7af768989d880e433168d2bb36618ca2a82" -> 404

It seeems that Jenkins gitlab plugin is calling /gitlab/api/v4/projects/test/test-project/repository/commits/b133f7af768989d880e433168d2bb36618ca2a82 instead of gitlab/api/v4/projects/gitlab_project_id/repository/commits/b133f7af768989d880e433168d2bb36618ca2a82

I thought maybe it could it be linked to the fact that my gitlab instance is running under /gitlab context instead of a more "classical" / context ? I didn't get any chance to investigate furthermore yet

[omehegan]

@mabrowning thanks for digging into this a bit more! When you refer to "an alternative regex," did you test that in the code? Or just confirm generally that it will match the valid strings which don't include .git? I've been intending to look for the regex and see if I could fix this, it looks like you've basically found it.

[omehegan]

@hervedevos can you let me know what versions of the plugin and of GitLab you are using?

[omehegan]

@mabrowning current regex: ^/?(?<projectId>.*)(\\.git)$
Your proposal: ^/?(?<projectId>.*?)(\\.git)?$

I think I'm not reading the regex quite right (my experience is all with Ruby regexes), but why do you add a ? after <projectId>.*? It seems like this would match on anything or nothing.

[hervedevos]

@omehegan I'm using Gitlab plugin v1.5.3 with Gitlab v10.4.3 and Jenkins v2.114

[omehegan]

@hervedevos your issue is something different. Please open a new issue for that, and attach more complete logs by following the instructions in the User Support section of the README.

[hervedevos]

@omehegan thanks, please have a look at #741

[omehegan]

@mabrowning can you respond to my comment at https://github.com/jenkinsci/gitlab-plugin/issues/691#issuecomment-378511708 ? I'm intrigued by your suggested fix, just want to make sure we get it right.

[mabrowning]

I did test that regex on ssh, http, and local paths. It works with and without .git. The
? in the .*? subexpression you asked about makes the * non-greedy, required for making the (newly optional) (\\.git)? pull the .git off the end. Otherwise, the .* has higher precedence and you end up with projectIds like group/project.git.

[omehegan]

@mabrowning thanks for explaining! It has just been a couple of years since I used regex regularly and my skillz are rusty. Let me do a patch release with this change and see if others watching this issue can test and confirm it resolves.

[omehegan]

Here's a patched version of the plugin with the regex change. If folks can test this and verify it resolves the issue, that would be appreciated.

http://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/gitlab-plugin/1.5.6-SNAPSHOT/gitlab-plugin-1.5.6-20180411.042719-1.hpi

[rnhurt]

@omehegan I just installed your patched version and it seems to work fine for non-multipipeline projects that don't have a ".git" at the end of the repository URL.

However, I'm having trouble with my multipipeline Gitlab URL that contains git@localhost:/.git and is considered malformed by this plugin. Using this URL (git@api-key-goes-here:user/jenkins-test.git) causes this plugin to fail on multipipeline jobs. @mabrowning's solution using ^/?(?<projectId>.*?)(\\.git)?$ seems to work, at least in my RegEx testing.

[omehegan]

@rnhurt I'm slightly confused - are you saying that the patched plugin works except for with multibranch jobs? But then you say @mabrowning's solution works?

[rnhurt]

Yes, the patched plugin works except for multibranch jobs. I was unable to fully test @mabrowning's solution but it seemed to be the correct RegEx.

[omehegan]

@rnhurt the patched version just includes @mabrowning's suggested change. Did the patched version break anything that was working? Or just not completely fix the issue? I'll have to look at what would be different in multibranch jobs.

[omehegan]

This fix was released in version 1.5.6 of the plugin.

[rnhurt]

@omehegan, the patched version did not break anything that I'm aware of. It just didn't fix my issue that I was having with multibranch jobs.

Thank you. :smile:

[omehegan]

@rnhurt if you think that issue is separate, could you open a new bug for it?

[binhnh17]

I fixed it by adding gibLabConnectionName to jenkinsfile (setting on jenkins dashboad is not enough):
options {
gitLabConnection('PERSONAL_ACCESS_TOKEN_TEST')
//'PERSONAL_ACCESS_TOKEN_TEST' => Token in jenkins credentials list mapping to personal access
//token generated from gitlab
}

[wangyachao-1111]

666

我已尝试将您建议的属性作为我的第一行。它具有相同的效果。我很确定这是我的开始，但我发现了一些stackoverflow和类似的帖子，表明其他属性设置已经解决了他们的问题。为了确保我刚刚尝试了两者，它仍然没有更新状态。

Following is my jenkins test log.

There are still pending GitLab builds. Please check your configuration
[Pipeline] // gitlabBuilds
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] gitlabCommitStatus
[Pipeline] {
Failed to update Gitlab commit status for project '1': HTTP 410 Gone
[Pipeline] sh

• echo hello
  hello
  [Pipeline] }
  Failed to update Gitlab commit status for project '1': HTTP 410 Gone
  [Pipeline] // gitlabCommitStatus
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  Finished: SUCCESS

[omehegan]

@wangyachao-1111 if you are using the latest version of the plugin (1.5.11), your issue is probably unrelated to this one. Please create a new issue, and attach your entire Pipeline script to it.

[tkruse]

For me using Cloudbees I also found it difficult to debug why status was not updated in gitlab. Just as a hint to anyone going through the same process:

• It matters what kind of Jenkins job you use (Freestyle, multibranch, ...). Don't assume they work the same.
• For multibranch, I needed the line properties([ gitLabConnection('my-correct-gitlab-connection-id')] ), and indeed it matters where it is put. For other kinds of jenkins job, it worked without. Don't assume anything about this.
• For multibranch, this would only work when using Mode: By Jenkinsfile in jenkins to read a script from SCM. With "Custom script", this would not work. Don't assume both options work the same.

I wish there were some error messages in the user logs (not system logs) to help debugging.