Shields: Badge request: Gitlab CI and Gitlab

[tankerkiller125]

It would be awesome if Shields supported Gitlab and Gitlab CI. They have a Decent API and it should be 100% feasible.

[espadrine]

Thanks for the suggestion! I believe they detail their API here, although I haven't looked into what kind of badges we can provide.

[tankerkiller125]

Unfortantly based on the documentation a authentication key is required no matter what. Would it be possible to get a key that Shields can use? Say for example a Shields account on gitlab just for this purpose?

[espadrine]

We use secrets.json for authentication (for instance, we use it with GitHub). As long as it isn't a per-user authentication, it works.

[tankerkiller125]

They only seem to be user specific for private projects (not a suggested thing to badge) internal projects and public projects can be viewed by anyone so long as they have an API key.

[ishitatsuyuki]

+1: any plans on this?

[moeabdol]

:+1:

[sergeysova]

Gitlab now supports CI badge!

[sergeysova]

[sergeysova]

Maybe add Gitlab issues count?

[gluons]

@LestaD But I still prefer Shields badge style. 😅

[paulmelnikow]

PR #841 has gone stale and needs work. If anyone would like to adopt it, please do!

[sum01]

The v4 API doesn't require authentication for most things. Unsure if that's what's stalling this or not.

[LVMBDV]

I wanted to poke around the API on https://gitlab.com before I began working on this and I must say it's weird. The documentation is lacking and in some cases, misleading (wrong?).

The docs for issue and merge request state that every API call to these functions must be authenticated but I can access them without any authentication on public projects. Try it yourselves:

curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/issues'
curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/merge_requests'

This is good for making badges but still, weird documentation.

Also, pipeline status(es) can be accessed only by the project owner, as per its documentation but there is a but. Despite the documentation (again), all commits of a public project can be accessed and the pipeline status of the last commit can be retrieved through the commits API. So we can make a badge for it after all.

Moreover, implementing a user token cycling subsystem like it was done for GitHub would be a security risk as it might allow other users to access private repository data due to the lack of a fine-grained permission system for application tokens (or personal access tokens). I haven't looked much into it but are we sure this is not a problem with private repositories on GitHub as well?

TL;DR We can make it happen but I wish GitLab had better permissions.

P.S. I hope GitLab isn't authenticating me through some black magic without a token. Please try the command above to confirm my findings.

[Snuggle]

```
~ ➜ curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/issues'

• Trying 52.167.219.168...
• TCP_NODELAY set
• Connected to gitlab.com (52.167.219.168) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
• successfully set certificate verify locations:
• CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Client hello (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS change cipher, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=gitlab.com
• start date: May 11 00:00:00 2018 GMT
• expire date: May 11 23:59:59 2019 GMT
• subjectAltName: host "gitlab.com" matched cert's "gitlab.com"
• issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
• SSL certificate verify ok.

GET /api/v4/projects/coldnight%2fci-test/issues HTTP/1.1
Host: gitlab.com
User-Agent: curl/7.60.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 14 Jun 2018 21:00:35 GMT
< Content-Type: application/json
< Content-Length: 7699
< Cache-Control: max-age=0, private, must-revalidate
< Etag: W/"71530dc535d8812ea57aeb5f6d419c91"
< Link: https://gitlab.com/api/v4/projects/coldnight%2Fci-test/issues?id=coldnight%2Fci-test&order_by=created_at&page=1&per_page=20&sort=desc&state=all; rel="first", https://gitlab.com/api/v4/projects/coldnight%2Fci-test/issues?id=coldnight%2Fci-test&order_by=created_at&page=1&per_page=20&sort=desc&state=all; rel="last"
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Next-Page:
< X-Page: 1
< X-Per-Page: 20
< X-Prev-Page:
< X-Request-Id: 16c93385-31dc-4c40-a829-4e8a68685956
< X-Runtime: 0.188505
< X-Total: 9
< X-Total-Pages: 1
< Strict-Transport-Security: max-age=31536000
< RateLimit-Limit: 600
< RateLimit-Observed: 1
< RateLimit-Remaining: 599
< RateLimit-Reset: 1529010095
< RateLimit-ResetTime: Fri, 14 Jun 2018 21:01:35 GMT
<
[{"id":11752626,"iid":12,"project_id":5168279,"title":"Epale","description":"Epale","state":"opened","created_at":"2018-06-07T01:37:54.903Z","updated_at":"2018-06-07T01:37:54.903Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2421176,"name":"Yamel Senih","username":"yamelsenih","state":"active","avatar_url":"https://secure.gravatar.com/avatar/e8270c029a61298fee5373903a889c8f?s=80\u0026d=identicon","web_url":"https://gitlab.com/yamelsenih"},"assignee":null,"user_notes_count":1,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/12","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10959931,"iid":11,"project_id":5168279,"title":"test issue","description":"il faut changer l'image d'accueil","state":"opened","created_at":"2018-05-22T12:19:21.217Z","updated_at":"2018-05-22T12:21:00.293Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2327051,"name":"Matthieu Jourdain","username":"mjourdai","state":"active","avatar_url":"https://secure.gravatar.com/avatar/4c59cb12bccffb8f6459e423f1607b4f?s=80\u0026d=identicon","web_url":"https://gitlab.com/mjourdai"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/11","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10299621,"iid":7,"project_id":5168279,"title":"SuperR(@superr)","description":"hello","state":"opened","created_at":"2018-04-13T08:08:24.121Z","updated_at":"2018-04-13T08:08:24.121Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/7","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10298307,"iid":6,"project_id":5168279,"title":"superr","description":"hello","state":"closed","created_at":"2018-04-13T06:35:12.496Z","updated_at":"2018-04-13T06:55:47.239Z","closed_at":"2018-04-13T06:53:57.240Z","closed_by":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"labels":[],"milestone":null,"assignees":[],"author":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"assignee":null,"user_notes_count":2,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/6","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":9712207,"iid":5,"project_id":5168279,"title":"My first sample issue","description":"Steps to repro:\n\nThis is my first sample issue.Please ingnore ![4hit_URL]\n(/uploads/6c5962b228919a9fe8f1bbd088c41c98/4hit_URL.jpg)","state":"closed","created_at":"2018-03-14T05:28:48.887Z","updated_at":"2018-03-14T05:36:17.614Z","closed_at":"2018-03-14T05:36:17.609Z","closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2105368,"name":"Prabhu","username":"prabhuamtex","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5f6b6125e41f4053da8839c780d2ae11?s=80\u0026d=identicon","web_url":"https://gitlab.com/prabhuamtex"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/5","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":9694027,"iid":4,"project_id":5168279,"title":"jjj","description":"","state":"opened","created_at":"2018-03-13T08:51:58.143Z","updated_at":"2018-03-13T08:51:58.143Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":727716,"name":"ganggang","username":"ganggang","state":"active","avatar_url":"https://secure.gravatar.com/avatar/8d2e85d5fd0de2a495fa7bc197918cb6?s=80\u0026d=identicon","web_url":"https://gitlab.com/ganggang"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/4","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8859575,"iid":3,"project_id":5168279,"title":"cryptocoopx-ccx","description":"","state":"closed","created_at":"2018-01-26T02:32:22.344Z","updated_at":"2018-01-26T02:32:26.864Z","closed_at":"2018-01-26T02:32:26.860Z","closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":1963760,"name":"Naif ALhaider","username":"Naif711","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5e77d108fa5a43b854cd75703a56e98f?s=80\u0026d=identicon","web_url":"https://gitlab.com/Naif711"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/3","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8859520,"iid":2,"project_id":5168279,"title":"cryptocoopx-ccx","description":"","state":"opened","created_at":"2018-01-26T02:24:43.413Z","updated_at":"2018-02-14T19:38:43.360Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":1963760,"name":"Naif ALhaider","username":"Naif711","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5e77d108fa5a43b854cd75703a56e98f?s=80\u0026d=identicon","web_url":"https://gitlab.com/Naif711"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/2","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8786090,"iid":1,"project_id":5168279,"title":"issur","description":"BUg","state":"opened","created_at":"2018-01-23T07:06:29.246Z","updated_at":"2018-04-13T18:11:38.445Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"auth* Connection #0 to host gitlab.com left intact
or":{"id":1953843,"name":"waqar ahmad","username":"waqar92","state":"active","avatar_url":"https://secure.gravatar.com/avatar/79f866cdc934803a60a2cfded672b430?s=80\u0026d=identicon","web_url":"https://gitlab.com/waqar92"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/1","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null}]% ```

[chris48s]

Thanks for doing some work on this @LVMBDV - appreciate that even answering the question "is there an API endpoint we can use for this badge?" can take some effort sometimes.

I hope GitLab isn't authenticating me through some black magic without a token. Please try the command above to confirm my findings.

I tried the commands you posted and I am getting a response back. Looking at the response headers, it contains keys:

RateLimit-Observed
RateLimit-Remaining
RateLimit-Reset
RateLimit-ResetTime

so it looks like unauthenticated use is rate-limited. Shields has the potential to generate quite high API traffic if we are proxying use for many users of a popular service, so rate limited access may not be suitable for this. We would probably need to authenticate to stay under rate limits/keep consistent service, even if authentication is not needed for repo permissions.

Additionally if the behaviour you intend to rely on does not match the documentation, maybe post on https://gitlab.com/gitlab-com/support-forum/issues to clarify. I think its better to ensure the behaviour isn't going to change out from under us before submitting a PR for a badge that relies on it.

[LVMBDV]

I guess we can do the GitHub token cycling thing with a GitLab application token that doesn't ask for any permissions (just authentication). I tried to create one and I was able to.

Additionally if the behaviour you intend to rely on does not match the documentation, maybe post on https://gitlab.com/gitlab-com/support-forum/issues to clarify.

I should, maybe tomorrow. Will post a link here when I do.

P.S. My authenticated queries came back with the same rate limit (600) so that's another bit of information.

[LVMBDV]

I made some progress but I would like to push them and open a PR after I clean it up a bit. I got these working without implementing token cycling yet:

• forks
• stars
• contributors

What I plan to also implement:

• last commit's pipeline status
• open & closed issues
• open & closed merge requests
• last activity date
• last commit date
• total commits
• repo size

And of course, token cycling.

[paulmelnikow]

Hey, I'm just tuning back into issues here after a little break.

Just a heads up that I rewrote and fixed the Github token cycling logic in #1205. It includes a
service-agnostic TokenPool with tests. It would be interesting to see how well the abstraction carries over to another use case. And it would be great to get some more eyes on that in general. Though sadly, I don't think I'll be comfortable merging it until I get access to server logs.

[LVMBDV]

The pooling token provider seems like a better solution to the problem at hand but I want to get GitLab support deployed as soon as possible with the smallest maintenance requirements possible.

I could wait for #1205 to be merged to resume working on GitLab badges built on its token pools or finish it with its own token pool helpers and leave refactoring it after #1205 is merged to someone else.

[paulmelnikow]

Yea, I don't see merging #1205 any time soon. At the same time, I don't like adding tech debt / deferred maintenance to get in new features. How about this: I'll open a new PR with the helper classes from #1205, and get that merged, and then you merge in master, and update this to use those classes? That way we're moving the ball forward on both fronts.

[LVMBDV]

That would work for me. Thanks.

[LVMBDV]

Now that token pools are in master, I will start working on GitLab badges. While I was waiting for it, I made a little badge service of my own in Crystal.

[jhgoebbert]

Just a short comment:
Using the project-name in the url fails as GitLab API requires the url to be encoded and shields.io eats that encoding:

https://img.shields.io/badge/dynamic/json.svg?label=issues&url=https://gitlab.com/api/v4/projects/coldnight%2fci-test&query=$.forks_count&colorB=brightgreen

It is required to use the project-id instead.

https://img.shields.io/badge/dynamic/json.svg?label=issues&url=https://gitlab.com/api/v4/projects/5168279&query=$.forks_count&colorB=brightgreen

[paternal]

Since http://gitlab.com is not the only instance of gitlab (see for example http://framagit.org), bonus points if URL of the gitlab instance can be passed as an argument to this badge.

Thanks for your work!

[opensorceror]

I second @paternal's comment. Since many organizations have an on-prem hosted instance of GitLab, it would be great to be able to specify a GitLab URL to an internally hosted Shields instance.

[aldomann]

It would be ideal if License badges were also supported (as they are in GitHub).

[paulmelnikow]

The very first GitLab badge went live today on https://shields.io/.

It works on gitlab.com and any public repo on public-facing GitLab instances. If you host your own GitLab inside a firewall you can run your own Shields inside the firewall, too.

This provides the same information as the default badge, with a few more formatting options.

This one was the easiest one to do first. More to come!

[gluons]

@paulmelnikow Thank you very much.

Now, I can use https://img.shields.io/gitlab/pipeline/%{project_path}/%{default_branch}.svg?style=flat-square for GitLab Badges on repo page. 😎

[noraj]

Look at what I did here: https://gitlab.com/rawsec/rawsec-cybersecurity-list/blob/master/js/sweetalert2.js

With Gitlab API, you can (while being UNAUTHENTICATED) grab forks number and stars number.

[noraj]

@paulmelnikow Nice to see GitLab badges were added. But pasting a gitlab project link like https://gitlab.com/rawsec/rawsec-cybersecurity-list on Suggests badges returns only the twitter badge. So one need to go in Build.

[paulmelnikow]

I just opened a new issue for the suggest issue.

@noraj Isn't there a rate-limit issue we'll run into on unauthenticated requests to the public API?

Note: #1838 was closed with some useful implementation bits which could be revived and ported to the new service architecture, once we sort out how to handle the authorization.

If someone wants to work on this, it would be very much appreciated! We have a pretty good toolset which is manageable even if you don't write a lot of JavaScript. The other maintainers and I are happy to lend support here or in Discord.

[paulmelnikow]

It's clear that Gitlab badges would create a lot of value for the community. I've reached out to Gitlab to ask if we can get an API key with a high limit so we can make these badges reliably.

Our workaround for GitHub is to create an OAuth app with a pool of many users' tokens. However this is a lot of work to set up and maintain! A mega-token for Shields (or a handful of static tokens) would be _much_ simpler!

It wasn't clear what the best channel was for this within Gitlab, so I sent a tweet to start: https://twitter.com/paulmelnikow/status/1118613535079194624.

If anyone knows what the best channel is, and/or connect me with a person or department who can help, I'd appreciate it!

[YoshiRulz]

FWIW private repos on GitLab.com can still use the built-in badges, but they only render if you're logged in (first-party cookies) and have permissions.

[Bl4d3s]

One Question: I'm currently trying to get the badge working with a public repo inside a group. It states only not found. How can I use this with a group project? (Both are public, https://gitlab.com/TheEmpire/Backend/)

Edit: Fixed it. Apparently there is an additional setting under Settings > CI/CD > General Pipelines to set the Pipelines public. With that set, all works well.

[paulmelnikow]

Would you like to add that to the badge documentation? It seems like it could be useful to others! We have something similar on eg the Discord badge.

[jhgoebbert]

you can find my example project for CI/CD (incl. badges) here:
https://gitlab.version.fz-juelich.de/vis/jusense-cicd

[Bl4d3s]

@paulmelnikow Will do so gladly. I looked at the discord service. Is it correct I will only have to provide a const string named documentation with the doc?

Edit: I went ahead and wrote a PR, I hope I got everthing right. See #4218

[asdoi]

Until now, I did not find any other badges than build status for GitLab. So I created a repository with examples, that use the basic API information without any access tokens: https://gitlab.com/asdoi/gitlab-badges
As there is now token required maybe some of them can be integrated in shields.io

[calebcartwright]

@asdoi - Thank you for sharing. However, there's still outstanding issues around Gitlab's API rate limits (which are IP based AIUI).

The hope was that Gitlab would be able to offer Shields an increased rate limit threshold if the Shields requests were authorized with a known/agreed-to account; it wasn't really about whether the data could be retrieved anonymously (GitHub supports anonymous API calls too, but Shields can't use them due to rate limiting).

IIUC the current thresholds on gitlab.com are 10 requests/second
https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits

Even during slow/non-peak windows, the Shields servers are issuing well over 1,000 requests/minute to the GitHub APIs, so Shields would almost certainly exceed the Gitlab rate limits given the popularity of the platform.

We won't be able to expand Shields' support for Gitlab badges until this rate limiting issue has been solved.

[noraj]

@calebcartwright Is there an issue open on GitLab tracker?

[calebcartwright]

I don't know of any @noraj. I know @paulmelnikow had started a dialog with the Gitlab folks but believe the fizzled out.

[ENDrain]

But does Shield have to use its own access token? Can user access token be passed to endpoint as a parameter, so that Shield makes an API request with user/group/whatnot token?