Shields: Getting spammed by shields.io

[sgrif]

This morning between 8:50 AM and 9:40 AM UTC, we received 3400 requests from shields.io for crates.io/api/v1/crates/rincon, a URL which will reply with a 404. These requests were coming in as frequently as 8 times per second. This is not a reasonable behavior for a service like this.

[RedSparr0w]

That url is not returning a 404 response for me:

{"crate":{"id":"rincon","name":"rincon","updated_at":"2018-04-15T10:18:37.627236+00:00","versions":[89008],"keywords":["database","driver","client","nosql","typesafe"],"categories":["database"],"badges":[{"badge_type":"appveyor","attributes":{"project_name":null,"branch":null,"id":null,"repository":"innoave/rincon","service":null}},{"badge_type":"codecov","attributes":{"branch":null,"repository":"innoave/rincon","service":null}},{"badge_type":"maintenance","attributes":{"status":"experimental"}},{"badge_type":"travis-ci","attributes":{"branch":null,"repository":"innoave/rincon"}}],"created_at":"2018-04-15T10:18:37.627236+00:00","downloads":10,"recent_downloads":10,"max_version":"0.1.0","description":"An ArangoDB driver for Rust","homepage":null,"documentation":"https://docs.rs/rincon","repository":"https://github.com/innoave/rincon","links":{"version_downloads":"/api/v1/crates/rincon/downloads","versions":null,"owners":"/api/v1/crates/rincon/owners","owner_team":"/api/v1/crates/rincon/owner_team","owner_user":"/api/v1/crates/rincon/owner_user","reverse_dependencies":"/api/v1/crates/rincon/reverse_dependencies"},"exact_match":false},"versions":[{"id":89008,"crate":"rincon","num":"0.1.0","dl_path":"/api/v1/crates/rincon/0.1.0/download","readme_path":"/api/v1/crates/rincon/0.1.0/readme","updated_at":"2018-04-15T10:18:37.627236+00:00","created_at":"2018-04-15T10:18:37.627236+00:00","downloads":10,"features":{},"yanked":false,"license":"Apache-2.0","links":{"dependencies":"/api/v1/crates/rincon/0.1.0/dependencies","version_downloads":"/api/v1/crates/rincon/0.1.0/downloads","authors":"/api/v1/crates/rincon/0.1.0/authors"}}],"keywords":[{"id":"database","keyword":"database","created_at":"2014-11-20T20:17:27.672220+00:00","crates_cnt":175},{"id":"driver","keyword":"driver","created_at":"2014-11-20T20:17:27.684776+00:00","crates_cnt":27},{"id":"client","keyword":"client","created_at":"2014-11-21T05:39:04.948095+00:00","crates_cnt":75},{"id":"nosql","keyword":"nosql","created_at":"2015-08-13T15:17:59.808884+00:00","crates_cnt":19},{"id":"typesafe","keyword":"typesafe","created_at":"2018-04-15T09:50:03.836487+00:00","crates_cnt":6}],"categories":[{"id":"database","category":"Database interfaces","slug":"database","description":"Crates to interface with database management systems.","created_at":"2017-01-17T19:13:05.112025+00:00","crates_cnt":89}]}

But i assume it has something to do with this repo,
not sure what would have caused the mass of request though.

Have the request stopped? was it just during that hour?

[sgrif]

The requests have stopped, yes. We still see occasional suspicious spikes to the same domain from shields, but nothing of that volume.

[RedSparr0w]

I wonder if their repo just had a recent spike in popularity which could have caused this.

It may be worth asking them to add ?maxAge=86400 or similar so the badge doesn't have to reload every refresh of the page.

_Note: the maxAge amount is in seconds (86400 = 24 hours)_

[SuperSandro2000]

Maybe the maxAge tag should be applied with a standard of 3 hours or something similar if it isn't specified.

[paulmelnikow]

Ref: #1723

[sgrif]

Regardless of caching, has there been any investigation performed into why so many requests were performed in such a short period? Has it been confirmed that it was due to the same number of requests coming into shields.io and not another issue?

[paulmelnikow]

Thank you so much for the detailed report. I'm really sorry for the traffic incident back in April.

Over the several months since then, we've addressed this through changes in our cache headers, the addition of a downstream cache, and adding Referrer-based rate limits. All of these things helped prevent issues like this from recurring.

A more targeted fix for issues like this is to cache API responses. #1575 is the best place to track future work on that.

Has it been confirmed that it was due to the same number of requests coming into shields.io and not another issue?

Yes. This is the only reasonable explanation.

Again, our apologies, and thanks.