Gitlab-plugin: False positive build status in declarative pipeline

Created on 24 May 2017 · 14Comments · Source: jenkinsci/gitlab-plugin

Context

Gitlab plugin version: 1.4.5
Gitlab version: 8.16.7
Jenkins version: 2.46.2
Job type: Multibranch Pipeline

Problem description

I initially thought it was specific to having a parallel step in the pipeline and one of the parallel steps passed but I still see it when I fail a regular stage with no parallel steps.

For example this pipeline:

def buildRevision = "LOCAL"
pipeline {
    agent {
        label "linux"
    }
    options {
        ansiColor('xterm')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        gitLabConnection('internal_Gitlab')
        gitlabCommitStatus(name: 'jenkins')
        timeout(time: 15, unit:'MINUTES')
        timestamps()
    }
    parameters {
        string(name: "releaseName", defaultValue: "RM_TEST", description: "Ivy release spec")
        string(name: "state", defaultValue: "testing")
    }
    tools {
        jdk 'jdk1.8.0_102'
        maven 'maven-3.5.0'
    }
    environment {
        MY_MAVEN_HOME = tool 'maven-3.5.0'
        CREDS = credentials("${params.state}-dummy-credential")
        RELEASE_NAME = "${params.releaseName}"
    }
    stages {
        stage("prep") {
            steps {
                sh "env | sort"
                script {
                    buildRevision = "${params.releaseName}.${env.BRANCH_NAME}.${currentBuild.startTimeInMillis}.${env.BUILD_ID}"
                }
                echo "buildRevision: ${buildRevision}"
                withEnv(["PATH+PACKER=${tool 'packer-1.0.0'}"]) {
                    sh "packer --version"
                }
            }
        }
        stage("false positive") {
            steps {
                parallel (
                    "par-1" : {
                        sh "echo bad"
                        sh "exit 1"
                    },
                    "par-2" : {
                        sh "echo good"
                    }
                )
            }
        }
    }
}

The build fails in the par-1 step as expected, but it posted a success status to GitLab. Replacing the false positive stage with

            steps {
                sh "echo bad"
                sh "exit 1"
            }

It is still posting a success to GitLab even though the Jenkins build failed.

I haven't tried replacing the global options gitlabCommitStatus with per-stage status.

doc improvement

Source

jgeorgeson

👍1

Most helpful comment

You are correct as to the point of the gitlabBuilds() in the options{}. Although I am seeing the same behavior as described in #554 (and have subscribed to it and the upstream GitLab issue) my current implementation seems to be functional. Simply updating the readme/docs would be, I think, enough to close this issue.

But as a feature request (which I can file as a separate issue if you like) it would be nice to see the gitlabBuilds() call setup to automatically (at lease with an optional true/false switch) send the running/success/fail/cancel statuses as each stage begins and completes, without an explicit call to gitlabCommitStatus() within each stage.

jgeorgeson on 7 Jul 2017

👍3

All 14 comments

I have to assume there's something wrong with my Jenkinsfile syntax because it just isn't posting a failure status when my build steps fail, only when I have invalid jenkinsfile syntax.

I added a log in Jenkins for GitLab plugin at level FINEST per your README, and this is what I see at the end of that log

Call GitLab:
HTTP method: POST
URL: http://gitlab.example.com/api/v3/projects/releng-dummy%2Fjenkins-dummy/statuses/490727655421478c7c6400b8a83f06efcb983a43
Request headers: [
Accept = [application/json],
Content-Type = [application/x-www-form-urlencoded],
PRIVATE-TOKEN = [****FILTERED****]
]
May 30, 2017 9:02:21 PM FINEST com.dabsquared.gitlabjenkins.gitlab.GitLabClientBuilder
Got response from GitLab:
URL: http://gitlab.example.com/api/v3/projects/releng-dummy%2Fjenkins-dummy/statuses/490727655421478c7c6400b8a83f06efcb983a43
Status: 201 Created
Response headers: [
Cache-Control = [max-age=0, private, must-revalidate],
Connection = [Keep-Alive],
Content-Length = [587],
Content-Type = [application/json],
Date = [Wed, 31 May 2017 02:01:33 GMT],
Etag = [W/"143fa380fad63ce310e1bb8a771b9a06"],
Proxy-Connection = [Keep-Alive],
Server = [nginx],
Strict-Transport-Security = [max-age=31536000],
Vary = [Origin],
X-Request-Id = [8935f285-97ac-477d-91b9-f8ef9f697ff7],
X-Runtime = [0.082914]
]
Response body: {
  "id" : 383661,
  "sha" : "490727655421478c7c6400b8a83f06efcb983a43",
  "ref" : "false-positive",
  "status" : "success",
  "name" : "jenkins",
  "target_url" : "http://jenkins.example.com/jenkins/job/jenkins-dummy/job/false-positive/9/",
  "description" : null,
  "created_at" : "2017-05-31T02:01:30.737Z",
  "started_at" : "2017-05-31T02:01:30.752Z",
  "finished_at" : "2017-05-31T02:01:33.434Z",
  "allow_failure" : false,
  "coverage" : null,
  "author" : {
    "name" : "CI Service Account",
    "username" : "builds",
    "id" : 6,
    "state" : "active",
    "avatar_url" : "http://gitlab.example.com/uploads/user/avatar/6/avatar.png",
    "web_url" : "http://gitlab.example.com/builds"
  }
}

I assume it's an innocuous omission in the logs but there's no POST data listed in this log output (the status, target_url, name/context, etc)

jgeorgeson on 31 May 2017

Putting the gitlabCommitStatus() inside the step stage seems to be working

        stage("false positive") {
            steps {
                gitlabCommitStatus(name: 'jenkins') {
                    sh "echo bad"
                    sh "exit 1"
                }
            }
        }

Further experimentation shows that per-stage use of gitlabCommitStatus() behaves more as expected, but it is less convenient to configure. Putting it in the global options{} (which is what the main README says to do) doesn't seem particularly useful as I can't trust that the actual build steps succeeded. So perhaps it's a declarative vs scripted issue in GitLabCommitStatusStep class.

jgeorgeson on 31 May 2017

Last night I thought I'd be ok with the per-stage status for now but thinking further, this leaves a window where someone could accept an MR that's showing a success (from an early stage of the pipeline) even though it hasn't run all stages yet. Or maybe not even a person, the MR could be set to automerge on successful build. @omehegan how are we supposed to get a singular running->pass/fail status posted for the whole pipeline?

jgeorgeson on 31 May 2017

I've tested and confirmed that an MR set to automerge on success will be merged at the end of the prep stage using this pipeline definition (can probably be stripped down to a more barebones pipeline to test, it's just what I had to start with for testing).

def buildRevision = "LOCAL"

pipeline {
    agent {
        label "linux"
    }
    options {
        ansiColor('xterm')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        gitLabConnection('internal_Gitlab')
        timeout(time: 15, unit:'MINUTES')
        timestamps()
    }
    parameters {
        string(name: "releaseName", defaultValue: "RM_TEST", description: "Ivy release spec")
        string(name: "state", defaultValue: "testing")
    }
    tools {
        jdk 'jdk1.8.0_102'
        maven 'maven-3.5.0'
    }
    environment {
        MY_MAVEN_HOME = tool 'maven-3.5.0'
        CREDS = credentials("${params.state}-dummy-credential")
        RELEASE_NAME = "${params.releaseName}"
    }
    stages {
        stage("prep") {
            steps {
                gitlabCommitStatus(name: 'prep') {
                    script {
                        buildRevision = "${params.releaseName}.${env.BRANCH_NAME}.${currentBuild.startTimeInMillis}.${env.BUILD_ID}"
                    }
                    echo "buildRevision: ${buildRevision}"
                    sh "sleep 60"
                }
            }
        }
        stage("work") {
            steps {
                gitlabCommitStatus(name: 'work') {
                    sh "echo \"Doing some work\""
                    sh "sleep 60"
                    sh "exit 1"
                }
            }
        }
    }
}

jgeorgeson on 31 May 2017

I think I can achieve what I need (ie a single comprehensive and correct build status) by having a post block for the pipeline which calls the gitlabCommitStatus() step. Is this how it's intended to be used?

jgeorgeson on 31 May 2017

Nope, that doesn't appear to be working out well either. What I have now, and I think is working, is this:

Use gitlabBuilds(builds: [...]) in options{} to mark all the jenkins pipeline stages as pending jobs for the gitlab pipeline.
Use gitlabCommitStatus() in each stage in jekins pipeline.

This seems to prevent the automatic merge on success from merging after the first stage succeeds. I find it less than ideal though as the statuses in Jenkins don't appear as stages in GitLab, (as is discussed in #554).

jgeorgeson on 31 May 2017

👍1

@jgeorgeson sorry for not responding sooner. I think the final solution you have arrived at is the right one. In fact, per https://github.com/jenkinsci/gitlab-plugin#gitlab-configuration I think if you just wrap your stages in a gitlabBuilds{} block, you will get the desired result. I don't see that it needs to be part of options{}. Do you think there is anything more we could do here? Improve the documentation?

omehegan on 6 Jul 2017

I can't just make the gitlab calls in the stages or GitLab won't know ahead of time there are more statuses to expect before the pipeline is complete. There reason is when you set an open MR to automatically merge on pipeline success, it will merge on the first stage without waiting for the rest. If there was some extra magic in the gitlabBuilds when used in the options, such that it automatically matches stage names and posted the appropriated status as each stage completes, that would be the most convenient. Especially if smart enough to still post a status for a stage whose when condition is not met. I currently have to have post that always runs to check the negative of each stage's when condition to close out the advertised build status.

jgeorgeson on 6 Jul 2017

👍1

@jgeorgeson well, what I was getting at is simply that you could put the gitlabBuilds block around your stages, rather than using it in options{}. Either way, as I understand it, the point of that block is to tell GitLab at the start, "Here are the stages we are going to run, consider these all to be pending." If that works for you in options{}, then I think you're doing the right thing. If those statuses don't render in an ideal way in GitLab due to #554, I think that's an issue on their side.

omehegan on 7 Jul 2017

jgeorgeson on 7 Jul 2017

👍3

@jgeorgeson can you confirm that I've documented this correctly? Now that I re-read it I think I am still missing something that you're saying needs to happen in the stages, or in the post block.

omehegan on 19 Sep 2017

Looking much better but slight correction. The new docs are only posting a status in the post (which is the simple case). In my pipelines I'm calling gitlabCommitStatus in each stage so that I can see in GitLab which stage failed (for example when I have a parallel build I know which parallel branch failed). The pre-declaration of each stage in the options is so that the there is still a pending job in GitLab until the final stage has completed. But the pre-declaration in options isn't needed if we only send a status in the post.

jgeorgeson on 30 Sep 2017

If there was some extra magic in the gitlabBuilds when used in the options, such that it automatically matches stage names and posted the appropriated status as each stage completes, that would be the most convenient. Especially if smart enough to still post a status for a stage whose when condition is not met. I currently have to have post that always runs to check the negative of each stage's when condition to close out the advertised build status.

@omehegan This extra magic could really improve the setup and usage of the plugin, as each stages could very easily be reported back to Gitlab. It was in fact, my expectation when I first read the documentation.