Danger: Expect danger to fail the jenkins build if there were any errors emitted

[jk]

Report

I'm not sure if I expect the right thing here, but my expectation was that when danger reports an error with the Pull/Merge Request, it should fail the build.

What did you do?

Set up a test repository on gitlab with a Dangerfile that has a fail directive. Then I created a new Merge Request on GitLab with changes which meet the condition to trigger that fail directive.

ℹ Please replace this with what you did.
I let Jenkins validate the merge request with danger. At the end of the build log I get this output:

+ bundle exec danger --no-ansi
grep: specs/: No such file or directory
grep: specs/: No such file or directory
Results:

Errors:
- [ ] Please add labels to this MR
- [ ] Please provide a summary in the Pull Request description

Finished: SUCCESS

What did you expect to happen?

Since danger emits errors, I expected it to fail the build.

ℹ Please replace this with what you expected to happen.

+ bundle exec danger --no-ansi
grep: specs/: No such file or directory
grep: specs/: No such file or directory
Results:

Errors:
- [ ] Please add labels to this MR
- [ ] Please provide a summary in the Pull Request description

Finished: FAILURE

What happened instead?

Build passed

Your Environment

• GitLab 8.10.x
• Danger 3.0.3
• Jenkins 2.20 installed on CentOS 6.5
• Which CI are you running on? Jenkins
• Are you running the latest version of Danger? No

[orta]

Interesting, yeah, this should fail in the GitLab provider.

It's more complicated in GitHub where it can use the API to fail the build but still allow other test scripts to continue (it would be frustrating to not run the normal CI tests simply because of not including a CHANGELOG )

I think for GitLab we should fail the process, aye

[jk]

Perhaps it's a good idea to make that configurable via a cli argument. So depending on your use case it can fail the build with an exit code != 0. (In my case danger is the last thing the build job runs)

[orta]

Agree, perhaps danger --fail-on-errors ?

[k0nserv]

The reason we don't update the build status for GitLab is because it requires Developer level permission FYI.

[k0nserv]

Could probably have it try to update the commit status and fail gracefully if it's forbidden. In case users in more controlled environments than gitlab.com wanna run danger with developer permissions

[orta]

^ This IMO is optimal

[jk]

I agree

Agree, perhaps danger --fail-on-errors

Nevertheless this should also be implemented. We have some folks here, where their main source of information about a problem with the software is their jenkins build monitor instead of a merge request status. For the concrete naming, I think fail on errors is fine.

[k0nserv]

So is this issue for adding the flag or posting the status? Can we close this issue and create two new ones instead?

[orta]

Sure, that works for me 👍

[k0nserv]

Opened #508 and #509 for this