Shields: GitHub badges are "unresponsive": rate limiting

[ducin]

This is what I see at shields.io now:

github shields seem not to work. I've tried to open them in a separate browser tab - the same result. But the owner/repo URL is correct.

[espadrine]

Thank you for raising this issue!

I have bad news. GitHub's rate limiting has two ceilings:

• 60 requests/hour for unauthenticated requests
• 5000 request/hour for authenticated requests.

We are using authenticated requests. Indeed, the headers I receive from GitHub include the following:

Status: 403 Forbidden
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 0

However, the rate limit reset timestamp does not seem to be correct, which suggests a bug in GitHub's rate limiting system. I am writing to them for clarification.

In the meantime, I will try to adjust the cache to avoid hitting the rate limit. If that is not enough, I will try to cache GitHub badges for a longer span of time. Ideally though, I would find a common ground with GitHub to increase the rate limit.

[RIAEvangelist]

I see the issue today as well. Any word back from them?

If not, I have an idea to allow additional requests : spin up multtple authenticated "apps" and round robin the requests through them all using the a shared cache on the back end.

An additional solution would be to have a backup cache every time the primary cache expired it would replace the entry in the long term cache (1 week?). If a request was made and rejected there would be a high likelyhood that stale info was available unless the package/repo was not very popular.

[espadrine]

GitHub has told me they have opened an internal issue for the API team to investigate and track this.

Having multiple apps fails, because they track IPs as well. Switching the server's IP would be wonky.

When we hit rate-limiting, we do use our LRU cache (when there is something in it). But the requests are very varied, and the cache gets filled within an hour. (Potentially, we could have a smarter cache than LRU, but tracking popularity sounds like it would require a lot of memory.)

[arcanedev-maroc]

Same here, i've got [vendor|unresponsive] for Github issues/PR badges.

https://github.com/ARCANEDEV/Breadcrumbs

[ajermakovics]

Hi all,
Perhaps Shields could serve a blank badge in these cases. Unresponsive is more informative but doesn't look that great in a project's readme. Just a thought

[vbauer]

@ajermakovics :+1:

[martincostello]

@ajermakovics :+1:

[kimwalisch]

This would be a very good workaround for the time being. And please put some pressure on GitHub, why haven't they come back to you in 2 months' time?!

[lilyball]

I don't even get 'vendor | unresponsive". I get a broken image when I look at shields.io, and when I try and load the badge URL directly, I get a CloudFlare "Error 504 Gateway time-out" page.

[espadrine]

@kballard Hmm. I added the 504 because of this discussion. The hope was to configure CloudFlare to cache everything, but only use that cache if I returned a 504. However, I can't find a way to configure CloudFlare in this way. I didn't think it would cause CloudFlare to return an HTML page instead.

[davisonio]

Any news on this?

[martinpaljak]

I wish Github provided itself such badges. People obviously like them.

[calidion]

+1
@martinpaljak

[DamienDennehy]

+1. I've had to remove my latest release shield and replace it with a static svg because GitHub shields were down so often in the last week.

[brunchboy]

Yeah, this is a huge bummer. The Github-related badges on my Github-hosted documentation essentially never work. I may have to just give up on them, but they provided nice information in a pretty way when they worked. Is there a forum where raising this issue might help motivate Github to fix things?

[martincostello]

Agreed. I've started commenting it out of README files as I've had to update them as the unresponsive badges don't give a good first impression.

[anko]

@brunchboy

Github doesn't have an official issue tracker, but I've posted this as a request on the unofficial tracker here. Their reply to my email is here: the suggestion has been passed on.

Github's only (but very responsive) official support channel is through email to [email protected]: if you have thoughts (or pressure) to add, I recommend asking there and linking to the above issue.

[espadrine]

Here's a message I sent to their support.

---

Hi everyone, and thanks for the great work you have been doing with this website!

I am the author, maintainer, and sysadmin for the shields.io project, and something of a coordinator for all vendors offering similar content, to ensure consistency.
I provide access to badges as colorful images providing efficient pieces of information.
A few projects on GitHub have started to use it, and I now count about 110M badges downloaded every month.

Over the existence of the project, people have wanted and contributed a few GitHub-specific badges, such as download information, tag / release / commit information, forks / stars / watchers / followers /issues counters, and licenses.
As people started using these, shields started hitting the upper rate limit.

That is in stark contrast with what happens with the traditional semi-official GitHub button (at https://ghbtns.com/), where the IP emitting the request is that of the browsing user, as the request is made in JS from an iframe. Yet, a badge such as this one would cost 85% less bandwidth in a quarter of the requests, and would fit in places which wouldn't allow the use of an iframe.

This brings me to the purpose of this mail: it would be beneficial to everyone if we figured out a way to provide the full power of shields.io badges without showing non-working badges from rate limits. I am absolutely open to any suggestion, and would love to discuss it with you. Given the donations I receive for the project, I have fairly low-speced servers, and they are not located in the US, which makes fetching information from GitHub possibly slower than it could be.

The corresponding issue on the shields.io project is located here, and a request was made on isaacs/github.

[brunchboy]

That is an excellent message. I hope they respond in the manner it deserves.

[espadrine]

Hi Thaddee,

Thanks for waiting while we discuss this internally. For now, we'd like to give you a temporary rate limit increase for the API so that you hit the limit less often and can provide a better experience to your users. We've bumped the rate limit for the Shields application (client_id _[redacted]_) to 12500 reqs/hour for the next 30 days. Can you please confirm this is the OAuth application the shields.io service is using for making GitHub API requests?

We'll also continue discussing this internally so that we can find a long term solution. We'll get back to you if the team has any questions for you or when we have any news.

Thanks again for providing a great service and for reaching out about this. I hope the better rate limit helps and let us know if you have any other feedback or suggestions you'd like me to share with the team.

Cheers,
Ivan

---

Hi Ivan,

Thanks a lot for bumping the rate limit! It will help a lot.

Can you please confirm this is the OAuth application the shields.io service is using for making GitHub API requests?

Yes, this is the right client_id.

We'll also continue discussing this internally so that we can find a long term solution. We'll get back to you if the team has any questions for you or when we have any news.

Absolutely! Don't hesitate to involve us at https://github.com/badges/shields/issues/529.

Cheers!
Thaddée.

[MeirionHughes]

presumably that 30 day limit-change has ended because I'm getting intermittent 404s for shield images.

[espadrine]

@MeirionHughes It is unrelated: https://github.com/badges/shields/issues/725. I'll expand on the situation tonight.

[espadrine]

Date: 2016-04-27

Hi Thaddee,

I hope you're doing well. After discussing this internally and considering options, we'd like to share our recommendation with you.

We don't believe we'll be running a version of shields.io internally in the near future (or of a similar service), and we can't offer a permanent rate limit increase for your service. Whatever rate limit we give you -- you'll probably be back in a month or two to ask for another increase because your userbase grew and you need to make more API requests. For that reason, we'd like to suggest an approach with which your total quota would scale with the number of users you have.

If you build an OAuth application (https://developer.github.com/v3/oauth/) and ask your users to authorize your application with their GitHub account -- the application will get an OAuth token for each of those users. The application can then use those tokens to make authenticate API requests (https://developer.github.com/v3/#authentication). Because the API rate limits for authenticated requests are per user (and not per application) -- your total rate limit would increase as more users use and authorize your application. This is actually the approach that most of our integrations take, such as Travis or CircleCI. Since you provide a useful service to your users -- I think it's reasonable for you to ask them to authorize your application and give it a token which it can then use to provide a service which is more reliable/available. For users who don't want to do that, you can use the client_id+client_secret combo for your application to satisfy as many of those requests as
you can.

Would such an approach work for you? To give you time to transition to this new approach, we'd be willing to offer an extension for the rate limit increase we gave you (12500 reqs/hour) for the next 60 days.

Best,
Ivan

---

Date: 2016-06-26

Hi Ivan!

Thanks a lot for the suggestion!

I will pool user tokens, which I will recommend users generate through the shields.io website. I will use those tokens in such a way as to minimize the risk of hitting a rate limit.

The two months you gave me were very helpful, given how busy I was at work. Thank you very much for that.

---

Date: 2016-06-28

Great -- happy to hear that approach works for you, Thaddee! Let us know if you have any other questions or if you run into any problems.

Cheers,
Ivan

[espadrine]

I have made a few tests and corrections over the past week, and I believe you can start using it. To increase the rate limit for GitHub-related badges, you can go to this page on the Shields.io website. Tomorrow, I will add that information as documentation on all GitHub-related badges (which shows up when you click on a badge on the front page).

[brunchboy]

That’s great! I have contributed a token, so hopefully it is helping in a small way. Thank you for working on this.

[espadrine]

Thanks a lot! It seems that each user key contributes 12.5k requests per hour, but that might be because of the custom rules they made for Shields. Even if it went down to 5k, thanks to the nine users that contributed, we are probably fairly safe now!

[brunchboy]

Excellent, and hopefully more people will see the documentation about how and why to do this, and continue to contribute.

[martincostello]

I just opened up GitHub Desktop and received this error message. Looks like the token I added has clobbered my entire API rate allowance. I was under the impression the limit was per-token/per-application. If not I'm afraid I'm going to have to revoke the token.

[brunchboy]

I was wondering if that might be an issue. If we can get enough people to contribute tokens, this should never happen though.

[espadrine]

I am awfully sorry, I substantially misinterpreted how the tokens work. I am working on a fix, don't hesitate to revoke the tokens in the meantime.

[martincostello]

No problem. I'll create a new one once the fix is deployed.

[espadrine]

There's a partial fix up. Shields no longer uses a token if it has reached 1/4th of its rate limit.

[espadrine]

The full fix is up. Shields will only use a user token that has more than 1/4th of its rate limit _and_ is the user token with the most requests left. As a result, it self-balances between all user tokens available.

[brunchboy]

That’s a nice, elegant solution. And ¼ is conservative enough that nobody should be afraid to share their tokens.

[espadrine]

Thanks a lot for raising the issue!

I can probably raise the "1/4th of the rate limit" restriction. It corresponds to 3125 requests in an hour, which is a bit low (for tinkerers). However, that limit is only there as a safeguard. Currently, with just nine tokens, the lowest a token went in the past hour was 10288 requests, before getting reset to 12500.

[martincostello]

I've added a new token back into the app.

[espadrine]

Hi everyone,

I made a mistake.

@agboom pointed out to me in private that the GitHub user tokens were publicly accessible at https://img.shields.io/.github-user-tokens.json. (Obviously, it is no longer the case.)

The fundamental problem is that, when I first created the service, there was no sensitive information in it, and I didn't expect to ever have some. So I simply set the repository directory as the fallback location wherein the server could serve files, and forgot that I did. That was a mistake in retrospect, as when I first hit the rate limit problems a year or two later, I simply assumed that files in the main folder would not be publicly accessible.

First, correction and mitigation. I set a specific folder to contain public files, so that the server is not allowed to serve files outside of that folder. I set a specific folder to contain sensitive files, in private/.

Second, assuming the worst. The two sensitive files that we currently have are .github-user-tokens.json and secret.json. They are both very similar: they contain information that prove that shields.io is the one performing a request, and therefore, that it can be trusted with a higher rate limit. (I am sorry to have been sloppy with that trust.)

Malicious actors can have downloaded those files, and benefited from higher rate limits in Bintray, SL Insight and GitHub. For Bintray and SL Insight, anyone can obtain their own tokens to get that same rate limit by registering as a user at those companies, so there is no practical reason for them to use mine: if they wanted to be malicious, they could create fake accounts. For GitHub, it is a bit worse, as about 900 GitHub users registered to help increase the rate limit for GitHub badges, creating a rate limit of 11250000 (~ 11 million) requests per hour: virtually unlimited access to the public GitHub API. For GitHub's sake, I cannot keep using them given that they are potentially compromised, potentially giving some malicious actor a way to circumvent GitHub's rate limit mechanism, so I must revoke them.

So here is what I did:

• I regenerated all secrets,
• I revoked all GitHub user tokens for the Shields app.

Unfortunately, at peak time, I do need about 200k requests/hour. It is way below the 11 million, but way above the 12.5k that I can get from just me. I need about 20 people to re-activate the Shields app on GitHub, which may not happen in time for Monday 5pm CET, which is a typical peak time.

For users, including those that authorized Shields as a GitHub app, there was no security issue, as I originally set the app to require the minimal amount of permissions (it can only see what is already public). There is a potential denial of service issue, as a malicious entity could have used up all of your rate limit capacity, which would have prevented you from using, for instance, GitHub for Windows, up until the end of the hour.

For Shields.io, the revocation of GitHub user tokens means that GitHub badges may start failing at the end of every hour at peak time on Monday, unless about 20 people re-activate the Shields app by going to this page.

I am awfully sorry for this, and will do my best to be more worthy of your trust in the future.

[martincostello]

@espadrine Thanks for the explanation and actions you've token once you discovered the problem. I have added a new token to the service for use.

[davisonio]

Thank you for letting us know this.

Added new token!

[mathiasbynens]

Kudos for being so transparent about this, @espadrine! You have my sword. And my bow. And my token.

[agboom]

Well mitigated @espadrine. Please accept my token as a token of gratitude.

[niemyjski]

Is this why https://img.shields.io/badge/donorbox-donate-blue.svg fails to load

[niemyjski]

Could you add a section to the top of the readme about what this is and how to donate your client token (even add it to the top of shields.io?

[davisonio]

@espadrine did you get the 20 required signups? If not I can tweet or something to help.

[espadrine]

There thankfully were enough tokens. However, the volume of requests at peak time gets fairly high now, and I will need to add a server and do some performance tuning. I talk about that here: #868.

[Daniel15]

Can this issue be closed? It sounds like everything is going well with the GitHub badges 😃

[brunchboy]

As far as I can tell!

[Daniel15]

Great! Let's close it out and reopen if any issues come up 😄

[ghybs]

Hi,

Thank you for having pro-actively revoked our tokens.

Just to let you know that the page to (re-)authorize GitHub Shields App no longer works:
https://img.shields.io/github-auth

I assume you currently have enough tokens for the rate limit?
Please let us know should you need more tokens.

[mitchellkrogza]

Just ran into this issue this morning using some new badges for my repo. Tried visiting https://img.shields.io/github-auth but only get GitHub OAuth authentication failed to provide a code.

[paulmelnikow]

Thanks, let's address at #1038.

Let's open a new issue for related Github problems, so it's not buried in here.