Danger: GitLab instructions difficult to follow

[glensc]

Report

Following this doc:
http://danger.systems/guides/getting_started.html#including-danger

1. i added gem 'danger-gitlab' to my Gemfile

2. easy mode says run bundle exec danger init, but that asks github specific questions.

3. self hosted gitlab, says "example: ", but there's no example.

for example i'm confused should the hostname be with schema prefix (https://gitlab.example.net) or not (gitlab.example.net), should the base url include hostname (https://gitlab.example.net/api/v3 or not /api/v3)?

1. creating token says go visit url to create token, but does not specify what scopes should be given (as of GitLab 9.4.0 the choices are api, read_user, read_registry).

[glensc]

also i do not understand how to use danger-ci with gitlab.

just following the instructions, i get:

1. from terminal

➔ bundle exec danger --verbose
Could not find the type of CI for Danger to run on.

1. from ci:

$ bundle exec danger --verbose
Not a Pull Request - skipping `danger` run

Then there's this PR comment https://github.com/danger/danger/pull/861#issuecomment-317365766 linking to https://gitlab.com/gitlab-org/gitlab-ce/issues/15280
from there i see there's workaround:

export PR_BRANCH=$(curl -s "http://domain/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.source_branch")

export PR_ID=$(curl -s "http://domain/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid")

however none of that helps, i verified that those vars get proper value, yet danger doesn't check those variables (PR_BRANCH, PR_ID), i grepped sourcecode.

[sleekybadger]

Hi @glensc . You need to set CI_MERGE_REQUEST_ID env variable, so this will be:

export CI_MERGE_REQUEST_ID=$(curl -s "http://domain/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid")

and with https://github.com/danger/danger/pull/861/files release, we will support GitLabCi natively

[glensc]

but there's still no doc what the other env vars are, and looks like that DANGER_GITLAB_HOST needs to be a hostname matching git remote, so current the definition "host that gitlab is running on" is rather vague.

1. my gitlab is running i host named soa.example.net
2. my gitlab web url is https://gitlab.test.example.net
3. my gitlab git clone url is [email protected]:glen/danger.git

what do i assign to DANGER_GITLAB_HOST value?

[glensc]

tried with #861 gemfile patch applied. my DANGER_GITLAB_API_TOKEN is definately set, it's visible in prior env command.

$ bundle exec danger --verbose

Could not set up API to Code Review site for Danger

For Danger to run on this project, you need to expose a set of following the ENV vars:
 - GitHub: DANGER_GITHUB_API_TOKEN
 - GitLab: DANGER_GITLAB_API_TOKEN
 - BitbucketServer: DANGER_BITBUCKETSERVER_USERNAME, DANGER_BITBUCKETSERVER_PASSWORD, DANGER_BITBUCKETSERVER_HOST
 - BitbucketCloud: DANGER_BITBUCKETCLOUD_USERNAME, DANGER_BITBUCKETCLOUD_PASSWORD

i need updated gitlab gem too?

➔ bundle exec gem list|grep dange
danger (5.3.3)
danger-gitlab (5.0.1)

but danger-gitlab-gem has no git tags published:
https://github.com/danger/danger-gitlab-gem/releases

[glensc]

even with seemingly versions you say that work, it does not

Using danger 5.3.3 from git://github.com/sleekybadger/danger.git (at feature/gitlab-api-v4@2e122d5)
Using danger-gitlab 5.0.2 (was 5.0.1) from git://github.com/danger/danger-gitlab-gem.git (at 8bff3e6@8bff3e6)

all three required vars are also present:

DANGER_GITLAB_API_BASE_URL=https://gitlab.test.example.net/api/v3
DANGER_GITLAB_API_TOKEN=XXX
DANGER_GITLAB_HOST=https://gitlab.test.example.net

[orta]

Aye, sorry, I wrote the documentation and I've + @sleekybadger never used Danger on GitLab - with responses like "does not work" it's hard to debugwhat the problem is, but I can definitely help update the documentation once you find it 👍

Can you show us some stack traces or output from Danger?

[viezel]

@glensc I might be able to help out. I agree, the docs are quite vague.

here is my setup:

stages:
  - test
  - review

variables:
  DANGER_GITLAB_API_TOKEN: "XXXXXXXX"
  DANGER_GITLAB_HOST: "git.example.com"
  DANGER_GITLAB_API_BASE_URL: "https://git.example.com/api/v4"

before_script:
  - export PULL_REQUEST_ID=$(git ls-remote -q origin merge-requests\*\head | grep $CI_COMMIT_SHA | sed 's/.*refs\/merge-requests\/\([0-9]*\)\/head/\1/g')
  - export PR_BRANCH=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.source_branch")
  - export PR_ID=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid")
  - export CI_MERGE_REQUEST_ID=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid")

test-sim:
  stage: test
  script:
    - pod install
    - xcodebuild clean -workspace MyApp.xcworkspace -scheme MyApp
    - xcodebuild test -workspace MyApp.xcworkspace -scheme MyApp -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.3.1' 
  tags:
    - ios

danger:
  stage: review
  script:
    - bundle install
    - bundle exec danger --verbose
  tags:
    - ios

I get:

$ bundle exec danger --verbose
Not a Pull Request - skipping `danger` run
Job succeeded

I dont have it working fully working, but at least I think im connected to my self hosted GitLab.

[Jeroenbb94]

At moment I have the same issue that danger is not finding my merge request a.k.a. pull request.

@viezel When I try your "exports" I'm receiving the following response:

$ export PULL_REQUEST_ID=$(git ls-remote -q origin merge-requests\*\head | grep $CI_COMMIT_SHA | sed 's/.*refs\/merge-requests\/\([0-9]*\)\/head/\1/g') $ export PR_BRANCH=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.source_branch") jq: error (at <stdin>:0): Cannot index string with string "sha" $ export PR_ID=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid") jq: error (at <stdin>:0): Cannot index string with string "sha" $ export CI_MERGE_REQUEST_ID=$(curl -s "https://git.example.com/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token=${OAUTH_TOKEN}&state=opened" | jq -r ".[]|select(.sha == \"$CI_COMMIT_SHA\")|.iid") jq: error (at <stdin>:0): Cannot index string with string "sha" $ bundle exec danger Not a Pull Request - skipping `danger` run Job succeeded

[viezel]

@Jeroenbb94 youre right. Its actually not working :(
Lets hope this new version will solve it.

[Jeroenbb94]

I'm using danger 5.3.4 that was released 2 days ago and danger-gitlab 6.0.0.

I've tried both gem 'danger' and gem 'danger-gitlab'. Both gave the same response "Not a Pull Request - skipping danger run while there is a merge request.

P.S. I've also tried the information that is provided in the PR https://github.com/danger/danger/pull/861, still no luck.

[orta]

I'd recommend making a fork of Danger and adding some debugging logs around the logic for Not a Pull Request - skipping danger as we don't have a gitlab setup, and so we can't get this set up ourselves to tests

[viezel]

@orta You should be able to run and test it out using the free gitlab.com account. They have free shared docker based runners for its CI.

[Jeroenbb94]

@orta Thanks for your response, I'll add some debugging logs tomorrow :)

[glensc]

as for the examples not showing up on webpage, i think the page renderer is broken, because the examples are present in source file:

• webpage
• source

i used github example (it's broken as well) because the javascript that shows/hides ci specific blocks does not support auto expand based on anchor from url.

i think the javascript "smart code" should be just removed, it's not worth it if it's so half-baked. just show all content on page.

[Jeroenbb94]

During debugging in gitlab_ci.rb I noticed that I had a CI_PULL_REQUEST_ID that matches my merge Request id from gitlab, thanks to the export CI_PULL_REQUEST_ID. But the CI_MERGE_REQUEST_ID was empty.

When I added:

- export CI_MERGE_REQUEST_ID=$(git ls-remote -q origin merge-requests\*\head | grep $CI_COMMIT_SHA | sed 's/.*refs\/merge-requests\/\([0-9]*\)\/head/\1/g')

The danger runned successfully.

P.S. This is the only export I'm using, I've removed the previous ones

[viezel]

@Jeroenbb94 thats great news.
How does your gitlab-ci.yml file look like?
with varialbes, exports etc ?

[Jeroenbb94]

@viezel Like this:

stages:
  - review

variables:
  DANGER_GITLAB_API_TOKEN: "XXXXXXXX"
  DANGER_GITLAB_HOST: "git.example.com"
  DANGER_GITLAB_API_BASE_URL: "https://git.example.com/api/v4"

danger:
  stage: review
  before_script:
    - export CI_MERGE_REQUEST_ID=$(git ls-remote -q origin merge-requests\*\head | grep $CI_COMMIT_SHA | sed 's/.*refs\/merge-requests\/\([0-9]*\)\/head/\1/g')
    - bundle install
  script:
    - bundle exec danger --verbose

[sleekybadger]

@Jeroenbb94 hm, interesting. We are doing the same already. What version of danger do you use?

[Jeroenbb94]

Version 5.3.4, fun fact: When I remove the export line, danger seems to work. I'm guessing it's a cache thing.

These are the versions I use:

Installing public_suffix 2.0.5
Installing claide 1.0.2
Installing colored2 3.1.2
Installing nap 1.1.0
Installing open4 1.3.4
Installing multipart-post 2.0.0
Installing git 1.3.0
Installing kramdown 1.14.0
Installing unicode-display_width 1.3.0
Installing multi_xml 0.6.0
Using bundler 1.14.6
Installing addressable 2.5.1
Installing cork 0.3.0
Installing faraday 0.12.2
Installing terminal-table 1.8.0
Installing httparty 0.15.6
Installing claide-plugins 0.9.2
Installing faraday-http-cache 1.3.1
Installing sawyer 0.8.1
Installing gitlab 4.2.0
Installing octokit 4.7.0
Using danger 5.3.4 

[sleekybadger]

Yeah, continues integrations often cache dependencies. I wanna try to run danger on self hosted gitlab, to check that everything good and to produce better setup guide. But I'll able to do this only on weekend ¯_(ツ)_/¯

[viezel]

@Jeroenbb94 I got it working too. I had to change the dangerbot user to reporter

[dsexton]

We submit merge requests from forked repositories, and in this case the project ID is that of our fork, and not of the main repository. This results in an empty list of merge requests when querying the project API. There are several issues on the gitlab project about this. (i.e. #23902)

As a workaround, you can hit the project API and get the upstream project id from the forked_from_project object. There is also a root merge request API coming in Gitlab 9.5.

[glensc]

btw, why set gitlab related variables manually? for example there exists CI_REPOSITORY_URL and CI_PROJECT_URL

https://docs.gitlab.com/ce/ci/variables/

[dsexton]

@glensc because CI pipelines belong to the project a given branch is pushed to, so if you have forked the project and push to your fork, then this ID differs from the project ID of the upstream repository.

When you submit a merge request to the upstream project, the merge request belongs to the upstream project, not your fork's project. Now we are in conflict. If you query merge requests via CI_PROJECT_ID, then you will receive an empty list, since the merge request is not yours, it belongs to upstream.

Furthermore, we have a timing issue here. As soon as you push a new branch, a pipeline is kicked off. If the danger job is ran before you submit your changes via a merge request, then they will be skipped. Your options are remembering to re-run this job, or (the route we took) create a "danger" binstub that will return a proper error code failing the job if no merge request id exists. We have this working on our on-prem instance of gitlab using a binstub to manually find the merge request ID.

[dvlprliu]

I finally get it working with gitlab-ci, here is my config:

# .gitlab-ci.yml
stages:
  - danger

Danger:
  dependencies: []
  stage: danger
  script:
    - gem install bundler
    - bundle install
    - fastlane danger_krn

# Fastfile
fastlane_version '2.40.1'

default_platform :ios

platform :ios do
  lane :danger_krn do
    ENV['DANGER_GITLAB_API_TOKEN'] = 'XXXXXXXXXXXXXXX'
    ENV['DANGER_GITLAB_HOST'] = 'mygitlab.com'
    ENV['DANGER_GITLAB_API_BASE_URL'] = 'https://mygitlab.com/api/v4'
    remotes = sh("git ls-remote -q origin merge-requests\\*head|grep #{ENV['CI_COMMIT_SHA']} || echo 'no pr' ")
    match_data = /.*merge-requests\/([0-9]+)\//.match(remotes)
    if match_data && match_data[1]
      ENV['CI_MERGE_REQUEST_ID'] = match_data[1] 
    else
      ENV['CI_PROJECT_ID'] = '<my upstream project id>'
   end
   bundel exec danger
  end
end

This will work with the fork workflow, because I set the upstream CI_PROJECT_ID manually, and the GitlabCI would find CI_MERGE_REUQEST_ID for me.

But the bsearch block implementation has a little issue, we should use >= instead of == according to the ruby doc, I've created a PR to fix this. https://github.com/danger/danger/pull/883

For the timing issue, I think it can be fixed by adding a trigger to the Danger job.

[okainov]

Any update here? It's still not clear and doesn't work out-of the box...

[orta]

Happy to take PRs improving it!

[Sarin1204]

For anyone still unclear on how to get this to work, I got it working by making danger run only on a merge request pipeline. https://docs.gitlab.com/ce/ci/merge_request_pipelines/

The gitlab env variable CI_MERGE_REQUEST_IID is only available on a merge request pipeline, which the danger gitlab integration requires.

So just add a only: [merge_request] key to your pipeline with the rest of the env variables mentioned (DANGER_GITHUB_API_TOKEN, DANGER_GITHUB_HOST, DANGER_GITHUB_API_BASE_URL) in the documentation and it should work.

[luisesantos]

Hey guys hopefully someone sees this message but is it considered safe to have the DANGER_GITLAB_API_TOKEN in the .gitlab-ci.yml file ?. It feels unsafe to me since this file is been checked in in the repo.

Thanks in advance

[Thomvis]

I believe the best practice is to add that variable through the Gitlab UI so it’s not checked in: https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui

[glensc]

I think it's fair to close this issue. It works for me now due further changes to library or changes in gitlab itself, not worth (my time) to dig that out now.

And secondly, people come to add "me too" to very long discussion is not helpful to anybody and seems neither of them is submitting PR to solve the original problem :D