Shields: img.shields.io is down

[paladox]

Hi when visting a link such as http://img.shields.io/travis/hexchat/hexchat.svg?style=flat it brings up the cloudflare error page also images shown on http://shields.io do not show. Please fix this.

[Foorack]

Yes, this have been going on for several hours, nearly all day local time zone. :-1:

[standage]

Wow, must be a serious problem to have over 8 hours of down time.

[marcwebbie]

Any news on that? I am experiencing the same issue for about 10 hours

[jacebrowning]

I am also experiencing this.

[montanaflynn]

@espadrine I'm getting a 523 response code from the img.shields.io host for all my badges:

$ curl -I https://img.shields.io/travis/montanaflynn/toy-spelling-corrector.svg
> HTTP/1.1 523 Origin Unreachable

[IonicaBizau]

Same here. Does somebody (still) take care of this project? :thought_balloon:

[egoist]

still down...

[ty0x2333]

has been lasting one day.@espadrine

[nhz-io]

2 days down

[marcwebbie]

What is the best shields.io alternative out there?

[espadrine]

Sorry, bad coincidence. I wasn't very available this week-end. I restarted the server computer… Linux crashed. It's usually really rare.

[IonicaBizau]

Shields.io works again! Thanks! :tada:

[marcwebbie]

Hello @espadrine

Thanks for fix :smile:

[IvanGoncharov]

@espadrine Thanks for fix.
You actually fixed 246,849 README files:
https://github.com/search?utf8=%E2%9C%93&q=filename%3AREADME.md+img.shields.io&type=Code&ref=searchresults

[Foorack]

@IvanGoncharov Wow, that's an incredible number.
@espadrine Thanks :smiley:

[IonicaBizau]

250 of these README.md files are mine. :joy: :grin:

[nhz-io]

thats it, time to make my own badger based on aws lambda

[swrobel]

Seems to be down again...

[IonicaBizau]

Same here... :sob:

[egoist]

This is as bad as GitHub goes down 🙄

[espadrine]

The server did not die, but it is getting a lot of load currently, way more than usually: something like 200 req/s. I'd suspect someone is playing with scripts and servers.

[espadrine]

I put CloudFlare in "under attack" mode for a few minutes, presumably breaking those scripts. The load has gone back to around 50req/s.

[nhz-io]

Just ban the origin

[nhz-io]

Another round of downtime

Its not down, but the time that it takes to respond with svg content is longer than githubusercontent will wait for data, so, its not down but its pretty much defunct

[standage]

@espadrine Are you still mitigating some kind of high request load or attack?

[nhz-io]

Its back up again. Its feels like waves of spam hitting the service

[levigroker]

Appears to still be highly unresponsive (down) as of last night and still today.

[tankerkiller125]

Probably related to #445

[IonicaBizau]

Down again! :disappointed: Isn't there a real fix for this behavior...?

[groggemans]

250.000+ README files are using shields.io, so that's a substantial part of the github users/repo's..
Why don't u ask the Github team to help you out by hosting this awesome project?
They should have the infrastructure to keep it working flawlessly.

[kailan]

What @groggemans suggested could be great. GitHub can clearly see the popularity of this site so I'm sure that hosting it or integrating it into the site would be at least considered.

[julmot]

Down again. If GitHub doesn't help you out try to use Cloudflare. I think availability is the most important in this project.
A fix is necessary asap, to promote this project with "We serve fast..." when serving nothing over days is ridiculous.

[foxylion]

@julmot They are already using Cloudflare.

[espadrine]

It's not done yet, but FYI, I'm configuring a new server. It will be DNS round-robin'ed. The peak requests has reached 60req/s, which despite my attempts at optimization seems to be overwhelming for a single core.

[kailan]

@espadrine If you need more capacity I'm more than happy to provide you with a few instances.

[espadrine]

All right, folks, we are now running two servers. One is in the Americas, and one is in Europe, but since which IP picked is random (I have two A (DNS) fields set up on CloudFlare to two distinct IP addresses), your location doesn't matter much. Things should get better as the world learns this DNS change.

I still have some work to do to improve monitoring and filtering against bad actors, but I'll call it a day for today.

[julmot]

:+1:

Hopefully this will allow faster requests. For me, the requests are taking longer than usual:

Also sometimes the server aborts the connection and only responses a few badges:

[julmot]

@espadrine I think many requests will come from the main page shields.io, because you are embedding all the images to show an example. Did you think about using base64_encode to reduce the request amount (currently about ~235 with huge response times)?

[espadrine]

@julmot That is absolutely something of a concern, although the plan is to load the images only when shown: https://github.com/badges/shields/issues/633. If you have time to do so, it would be a delight!

[julmot]

For me in the latest FF all images will be loaded on start.
I'll have a look into this in the next week.
Cheers.

[foxylion]

Maybe it would also be an idea to increase the cache heavily, especially the cache time. Digging into the code showed to me it is only caching 1000 badges with a 5 second cache time. Maybe increase it to 20.000 and 60 seconds? I think an update every 60 seconds should be fine for nearly all badges. As long as the bandwidth is not a problem, it could reduce the load. Do you have a metric how many cache hits you currently have?

Edit: When you have already Cloudflare in front of the application, you could also consider disabling the cache completely and relying on the Cloudflare cache (send the right headers).

[espadrine]

Maybe increase it to 20.000 and 60 seconds?

It used to cause too many OOM-killer crashes to be manageable, but it would probably be fine now, yes.

Do you have a metric how many cache hits you currently have?

I don't, although that's a longstanding TODO.

Edit:

Edit: When you have already Cloudflare in front of the application, you could also consider disabling the cache completely and relying on the Cloudflare cache (send the right headers).

I have tried to do so, but could not find a way. CloudFlare seems to have a cache targeted at regular websites; it will cache HTML pages and their associated content, but not arbitrary images with arbitrary rules.

[foxylion]

By default Cloudflare should not cache any HTML-Contents, but the images. (see here).

Note: CloudFlare, by default, does not cache HTML content. You need to write a Page Rule to cache static HTML content.

The underlying problem is (as far as I can see) that it looks like all the headers you send are targeting that the CDNs (Cloudflare, GitHub-CDN) does not cache anything.

[espadrine]

@foxylion What I had tried is this.

The minimum caching expiration available is 2h by default, and 30min through page rules. Don't you fear that people will find that to be too long?

[groggemans]

It depends on the badge. Things like a gitter chat badge can be cached for a very long time, a build status is a whole other thing. Maybe you could add a subdomein that provides longer caching? And keep the things that need short caching times like it is now?

[lfender6445]

i think service is out

[foxylion]

@espadrine Sorry, I didn't know that the lower times can only be used in the payed service of Cloudflare.

I would suggest another solution:
Provide a service (if it does not already exists) where the complete configuration of a badge can be encoded in the url. This service should also send the caching headers, maybe for a day or more as the resource should unlikely change.
The url pattern could be /badge/{style}//{color-right}/{text-left}/{text-right}/badge.[svg|png|gif]. A example would be /badge/flat/fff/License/Apache%202.0%2F2.1/badge.svg.

Then refactor all existing services to Respond with 302 Found status code, disable cache headers and redirect to the static badge URL.

Now you can enable the permanent caching of /badge/ on the Cloudflare side and every badge will only be generated once. This should force the browser to do the the default request e.g. /travis/user/repo.svg everytime, but then skip the actual request for the badge data itself.

The cache in your server could be changed to only cache the badges contents (color, text) for a given and prevent calling the remote APIs too often. It should also reduce the cache size drastically.

The full solution should reduce both server load and network traffic.

[standage]

@foxylion Works for badges that don't change, like license or gitter, but doesn't work for badges that read, for example, build status or version number from external services. And in my experience, that's what I see the vast majority of these badges being used for.

[foxylion]

@standage I think it should work. Cloudflare should only cache the static badge (which has erverything encoded in the url). The dynamic badges will do a HTTP redirect to the static badge. So you reference the dynamic badge url in your README and it will resolve to a static version of the badges current state.

The only problem I can see is that the GitHub CDN does too aggresive caching and caches the redirected image under the dynamic badge url, instead of serving the redirect.

Maybe an example to clarify the idea:
Request: /travis/user/repo.svg?style=flat
Response (not cached by Cloudflare): 302 Found, No-Cache, Location: /badge/flat/00f/Buildstate/Passing/badge.svg
Request: /badge/flat/00f/Buildstate/Passing/badge.svg
Request (cached by Cloudflare): 200 Ok, Cache, Badge as SVG

[standage]

@foxylion Code coverage is an interesting case: you don't want on the order of a hundred (0%-100%) static cached badges for each project, although with a smart approach I suppose it would be possible to reduce redundancy (finite number of possible percentage values and styles).

[foxylion]

@standage You won't cache anything. That does Cloudflare for you. ;-)
So each badge is generated exactly once and afterwards cached by Cloudflare. In the best case, forever.

[espadrine]

Provide a service (if it does not already exists) where the complete configuration of a badge can be encoded in the url.

That seems roughly what shields.io's existing /badge/ services does: http://shields.io/#your-badge.

I'm not sure why you're suggesting this, though. I think you assume that most of the CPU time is spent generating the images. However, that generation is a function call that does a string concatenation, which is pretty cheap. I probably could try to see how much CloudFlare would agree to cache, but I'd have to carefully monitor how many requests I get, as depending on CloudFlare, it could easily rise. In any case, I don't see how it could improve anything. In current requests, knowing when I need to call the vendors, calling them and treating the result is the expensive part.

[foxylion]

However, that generation is a function call that does a string concatenation, which is pretty cheap.

Okay, sorry then, when the traffic is not an issue that won't be a proper fix.

Maybe then just cache the responses of the APIs for a longer time.
Do the SVG generation always (if that is really near to 0 cost), so it will reduce cache size immensively.
And add a cache timeout per service to improve the performance even more.

probably could try to see how much CloudFlare would agree to cache, but I'd have to carefully monitor how many requests I get, as depending on CloudFlare, it could easily rise.

Monitoring the cache amount is quite easy, you can see the percentage in your Cloudflare control panel.

[clifg]

+1 -- I haven't been able to get any badges from the site today. All examples are currently failing (523) on www.shields.io as well.

[espadrine]

@clifg s0, our first server, died. The DNS round-robin seems to work in such a way that clients which point to a dead server take some time to learn to ask the live one. I'm learning more about DNS every day. I restarted s0.

[espadrine]

Haha, and a few minutes after I turned it back on, OVH's built-in DDoS prevention system flagged the sudden incoming flood of requests. Many requests were blocked for 15min. Back online.

What a wonderful Saturday morning.

[akyoto]

@espadrine Thank you!

[tankerkiller125]

At this point I think it might truly be in our best interest to get github to host it... They have more resources than what this project could possibly use. Not to mention unlike us they have money and vast networking capabilities.

[clifg]

@espadrine thanks! I appreciate you trying to keep this thing afloat. Also agree that it would be awesome if github would host it since so many projects use it.

[astrojuanlu]

Folks, it's great that everyone made their suggestions and all but this project receives less than $ 5 a week.

https://gratipay.com/shields/

I don't know what's the sustainability status of shields.io, maybe @espadrine can tell.

[304NotModified]

Related: https://github.com/badges/shields/issues/655

[kkodev]

It keeps failing again with 520 over last few days.
I'd be happy to host my own version, but it doesn't install anymore - #653 :(

[espadrine]

Currently, the DNS points to two IPs, which both seem to work fine:

• https://192.99.59.72/try.html
• https://51.254.114.150/try.html

(The self-signed certificate is normal, it's the procedure for using CloudFlare's Universal SSL.)

I have never heard of code 520. Is it 502 instead? Could it be caused by a proxy?

[kkodev]

https://img.shields.io/badge/2.0.0-40-blue.png doesn't work for me right now.
You will get it to work eventually, but just modify the numbers and you get the error again. Don't think it's a proxy issue.

[304NotModified]

PS last link works for me

[304NotModified]

And now also 520

Error 520 Ray ID: 29e601d97efc0c05 • 2016-05-05 17:45:38 UTC
Web server is returning an unknown error

[espadrine]

@kkodev Ah, that is a lot more specific, and I think I know why it doesn't work. While I attempt to fix it, I would recommend the use of SVG images instead.

[kkodev]

Thanks a lot man!

[304NotModified]

If you are the owner of this website:
There is an issue between CloudFlare's cache and your origin web server. CloudFlare monitors for these errors and automatically investigates the cause. To help support the investigation, you can pull the corresponding error log from your web server and submit it our support team. Please include the Ray ID (which is at the bottom of this error page). Additional troubleshooting resources.

[espadrine]

It ought to be fixed, although I can't recommend using SVG enough.

[espadrine]

Further maintenance was necessary on the Canadian server. It should now work across the board.

[kkodev]

Will certainly look into SVG as an alternative but for now being held up by https://github.com/fastlane/fastlane as a dependency:(
Thanks for help anyway, much appreciated!