Mastodon: Mastodon can be used as a DDOS tool

[valentin2105]

Hi !

Today I found this tweet : https://twitter.com/mattiasgeniar/status/892446659245993984

I tried to post a link on my instance, Mastodon.cloud, and follow the link's web server logs ->

400 instant requests.

Imagine I flood 10 link, I think that go to generate more than 4k requests..
It's not great for link's web server..

Any idea how to mitigate this on futur releases ?
Why Mastodon need to crawl the link ?

Thanks

---

• [ ] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [ ] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[angristan]

Tested on my account with 1.5k followers (I don't know how many instances).

Result : 578 requets in 20 seconds.

Examples :

2001:41d0:c:ec9::2b43:1d3 - - [01/Aug/2017:23:35:16 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://mstdn.io/)"
2001:41d0:c:ec9::2b43:1d3 - - [01/Aug/2017:23:35:16 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://mstdn.io/)"
62.210.247.240 - - [01/Aug/2017:23:35:16 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://oc.todon.fr/)"
163.172.20.35 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.3; +http://mastodon.partipirate.org/)"
2001:41d0:d:2389::7 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://mastodon.top/)"
2001:41d0:c:ec9::2b43:1d3 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://mstdn.io/)"
163.172.185.23 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0rc3; +https://mastodon.social/)"
62.210.247.240 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://oc.todon.fr/)"
163.172.20.35 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.3; +http://mastodon.partipirate.org/)"
2001:41d0:d:2389::7 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://mastodon.top/)"
2001:4b99:1:2:216:3eff:feaf:f948 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://m.massy.city/)"
188.165.228.227 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://mastodon.eliotberriot.com/)"
2001:bc8:3862:4272::42 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://mastodon.xyz/)"
163.172.185.23 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.5.0rc3; +https://mastodon.social/)"
2a01:4f8:10a:3e15::2 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.4; +http://framapiaf.org/)"
62.210.247.240 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://oc.todon.fr/)"
163.172.20.35 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.3; +http://mastodon.partipirate.org/)"
2001:41d0:d:2389::7 - - [01/Aug/2017:23:35:17 +0200] "GET / HTTP/1.1" 200 36402 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://mastodon.top/)"
2001:41d0:a:fb29::1 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://toot.tzim.net/)"
195.154.133.191 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://soc.ialis.me/)"
163.172.149.63 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://m.dumez.info/)"
2001:41d0:1:9064::1 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://antisocial.narinimous.fr/)"
91.121.37.224 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.7; +http://mastodon.at/)"
78.205.21.170 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0rc2; +https://ma.zy.lc/)"
2001:41d0:2:d230::15 - - [01/Aug/2017:23:35:17 +0200] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.5.0; +https://hostux.social/)"

[ykzts]

duplicate of #3518

[ghedipunk]

I wouldn't call it a duplicate.

Related, absolutely. Not duplicate.

The issue with #3518 is that a single server is making 3 different requests for the same resource.

The issue in this is that hundreds of servers are requesting the same resource, as soon as they get a notification of the toot.

[valentin2105]

Imagine you got 10 account on 10 instances and you post 10 links on each.
It would be a big mess.

[Gargron]

I can totally see this being a problem, but I also don't see a solution to it, because people want previews. An idea could be to randomly delay the fetching of the preview, however, you would get complaints from people "why doesn't the preview show up?" when they immediately check it. Anyone have any other ideas?

[ghedipunk]

Perhaps "trust, but verify." Also, humans are a great source of randomness.

Bear with me as my assumptions about how the federation works may not be completely accurate -- the details shouldn't matter.

Have the originating server send metadata about the link's preview along with the toot.

When the client requests the preview, their own server can then verify the link and update the preview if necessary.

This way, there's always something to show up front. Since the Web client doesn't attempt to show the preview until the user clicks on a toot, we have a random wait -- on the order of seconds, minutes, or years -- so we aren't creating a thundering herd based only on the toot hitting the federated servers.

Also, if nobody is even going to look at the preview, one never gets loaded, saving a few completely unnecessary requests.

The only real concern is a malicious server sending incorrect information in the toot's metadata... That's why I say to verify.

(This would also give users the ability to set a preferred preview image, as they're able to do on FB and G+, but that really belongs in a feature request rather than a discussion about a bug report)

[Hexalyse]

I was thinking about the same thing : couldn't the preview be generated when the toot is posted, by the server from where it's posted... then other instances would fetch this preview from the originating instance, as they do for media attachments ?

Just a random idea tho. I don't know if it would fit in the mastodon federating scheme.

[bortzmeyer]

Calling 400 HTTP requests a "dDoS" seems very exaggerated.

[ghedipunk]

Someone running Wordpress on a stripped down VPS without any CDNs mitigating traffic spikes would be very unhappy to see 400 HTTP requests hit in an instant.

Many hosts automatically reboot VPS servers if they exceed their memory allotment, bringing the site down for minutes... and on Mastodon, the first few minutes after a link is posted accounts for 95%+ of the visits that the site is going to get from that toot.

Arguably, the people who buy the cheapest VPS are the ones who need that traffic the most; by rebooting their server each time someone links to them on Mastodon, you're not just DOSing them, you're creating a bad user experience, driving away potential customers.

A DDOS doesn't have to last for days, harnessing the power of millions of misconfigured routers and IP cameras to be effective... It just has to come from more than one source at once and deny normal users use of the service.

[bortzmeyer]

@ghedipunk If I had to run a site on a "stripped down VPS", I would run a static site, for which 400 requests is nothing, even on a Raspberry Pi. But my main point is that generating 400 requests is very simple for any attacker. Mastodon is not really needed.

[LogalDeveloper]

This is not an issue just about 400 HTTP requests. The main issue here is that as Mastodon gets bigger and more instances are created, the larger and more serious the attacks can become. Another part of this is that while the attacks can be on purpose by a malicious person, they can also be on accident by an innocent person with a lot of followers who simply wants to share a meme they liked a lot.

[ghedipunk]

You're experienced in web hosting, as are the vast majority of people who would participate in this thread here on Github.

However, looking at small business sites, it's very clear that we're in the minority.

We know about Cloudflare and using nginx as a TLS terminator in front of Varnish-Cache that does intelligent edge-side includes on Apache/PHP generated content...

We know what design decisions to make on a completely static HTML only site so that if we need to update the site's layout, we can do it quickly.

We know that if you want to sell things online, you have to resize your 48000x28000 pixel product images _before_ displaying them to the users.

And, we know that there are users who don't know these things.

This isn't about helping us, the technical elites who can run a successful site off of a Raspberry Pi. It's about being good neighbors.

[mherrb]

I really don't see where the problem is. The distributed nature of mastodon will spread the requests over a (small for humans but large for the network) interval and when you publish some link on a public media, you expect traffic coming back to the site in question. Otherwise there's little interest on publishing it.
And when the number of mastodon instances will grow, the load will also be spread over a wider interval since it will take more time for the toot to propagate to all instances.

[bortzmeyer]

Indeed, when you post a link on Twitter, you get more requests (not from Twitter itself, because of its centralized nature, but because of all the bots that read Twitter and act).

[snarfed]

i noticed this recently too. >1k requests in <45s, >25qps. not a disaster, my site handled it fine, but still, noticeable. small thread on it here: https://mastodon.technology/@snarfed/100119606571241751 , cc @ashfurrow @neekz0r.

(qps numbers in the graph are artificially reduced due to a 60s+ aggregation interval.)

[neekz0r]

Haha.. the joys of having the same handle on multiple platforms.

Yeah, to me I think this should be considered a little more serious because theoretically this can be used as an amplification attack.

[MightyPork]

Here is my capture of this phenomenon if anyone's interested (removed IP addresses). There are a few real users, but mostly it's mastodon servers.

Click to expand

19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://glitch.social/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://vulpine.club/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.infra.de/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://toot-lab.reclaim.technology/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://wandering.shop/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://octodon.social/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tchncs.de/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.macsnet.cz/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://niu.moe/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://dev.glitch.social/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://touhey.org/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.social/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://metalhead.club/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.host/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://hackers.town/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.zombocloud.com/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://birb.site/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.at/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.io/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://oldbytes.space/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.xyz/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chaos.social/)" 
19:45:41 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.mit.edu/)" 
19:45:41 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.infra.de/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://glitch.social/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://toot.website/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.io/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://freedom.horse/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://metalhead.club/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://octodon.social/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://touhey.org/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.macsnet.cz/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://vulpine.club/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.social/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://niu.moe/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chaos.social/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tchncs.de/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.at/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://oldbytes.space/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.xyz/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.host/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://social.wxcafe.net/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://birb.site/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://toot.website/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0-tp0.1; +https://tootplanet.space/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://social.wxcafe.net/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://wandering.shop/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://hackers.town/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.zombocloud.com/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://dev.glitch.social/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://asonix.dog/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.mit.edu/)" 
19:45:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://freeradical.zone/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://toot-lab.reclaim.technology/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0-tp0.1; +https://tootplanet.space/)" 
19:45:42 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://freedom.horse/)" 
19:45:43 +0000] "HEAD / HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0-cybre; +https://cybre.space/)" 
19:45:43 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://asonix.dog/)" 
19:45:43 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0-cybre; +https://cybre.space/)" 
19:45:43 +0000] "GET / HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://freeradical.zone/)" 
19:46:08 +0000] "GET / HTTP/1.1" 200 65 "https://cloudwithlightning.net/secrets-and-pies" "bad horse" 
19:46:20 +0000] "GET / HTTP/1.1" 200 65 "https://cloudwithlightning.net/secrets-and-pies" "bad horse" 
19:46:24 +0000] "GET / HTTP/1.1" 200 65 "https://cloudwithlightning.net/secrets-and-pies" "bad horse" 
19:47:07 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 
19:47:07 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 
19:47:08 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 
19:47:46 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://glitch.social/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://dev.glitch.social/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tchncs.de/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.zombocloud.com/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://birb.site/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://vulpine.club/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.at/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://wandering.shop/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.mit.edu/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.io/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://niu.moe/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://toot-lab.reclaim.technology/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://toot.website/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.social/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tchncs.de/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://birb.site/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://touhey.org/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://hackers.town/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.infra.de/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://glitch.social/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.xyz/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.io/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chaos.social/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0-tp0.1; +https://tootplanet.space/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0-cybre; +https://cybre.space/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://metalhead.club/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.macsnet.cz/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://oldbytes.space/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://niu.moe/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.at/)" 
19:47:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.host/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.social/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://toot.website/)" 
19:47:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.zombocloud.com/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://vulpine.club/)" 
19:48:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://freedom.horse/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0-cybre; +https://cybre.space/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://touhey.org/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.mit.edu/)" 
19:48:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://freeradical.zone/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://oldbytes.space/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://metalhead.club/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://wandering.shop/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.infra.de/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.macsnet.cz/)" 
19:48:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://social.wxcafe.net/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.host/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chaos.social/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.xyz/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://toot-lab.reclaim.technology/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://social.wxcafe.net/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://dev.glitch.social/)" 
19:48:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://octodon.social/)" 
19:48:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://asonix.dog/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://sprite.land/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://octodon.social/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://hackers.town/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0-tp0.1; +https://tootplanet.space/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://freedom.horse/)" 
19:48:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://freeradical.zone/)" 
19:48:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://asonix.dog/)" 
19:48:49 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://retro.social/)" 
19:48:49 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://retro.social/)" 
19:48:49 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://social.literati.org/)" 
19:48:50 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://retro.social/)" 
19:48:50 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.4.1; +http://mastodon.club/)" 
19:48:50 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://social.literati.org/)" 
19:48:50 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/1.4.1; +http://mastodon.club/)" 
19:48:50 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mst3k.interlinked.me/)" 
19:48:50 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lou.lt/)" 
19:48:50 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lou.lt/)" 
19:48:50 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lou.lt/)" 
19:48:51 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lou.lt/)" 
19:48:51 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/1.4.1; +http://mastodon.club/)" 
19:48:51 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mst3k.interlinked.me/)" 
19:48:51 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://masto.donte.com.br/)" 
19:48:51 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.technology/)" 
19:48:51 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://awoo.space/)" 
19:48:51 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://masto.donte.com.br/)" 
19:48:52 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://tabletop.social/)" 
19:48:52 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.technology/)" 
19:48:52 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://awoo.space/)" 
19:48:52 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://tabletop.social/)" 
19:48:52 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://snabeltann.no/)" 
19:48:52 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://snabeltann.no/)" 
19:48:52 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://znark.us/)" 
19:48:53 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://witchey.club/)" 
19:48:53 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://snabeltann.no/)" 
19:48:53 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://witchey.club/)" 
19:48:53 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://sn.angry.im/)" 
19:48:53 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://znark.us/)" 
19:48:53 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://sn.angry.im/)" 
19:48:53 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://pentacl.es/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/1.6.1 +27 (bf7ce63); +https://is.a.qute.dog/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://kelnet.social/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://trunk.mad-scientist.club/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://elekk.xyz/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://trunk.mad-scientist.club/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://trunk.mad-scientist.club/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://opensocial.africa/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://trunk.mad-scientist.club/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://opensocial.africa/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://sn.angry.im/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://pentacl.es/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://elekk.xyz/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.2; +https://kelnet.social/)" 
19:48:54 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.sdf.org/)" 
19:48:54 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/1.6.1 +27 (bf7ce63); +https://is.a.qute.dog/)" 
19:48:55 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://bsd.network/)" 
19:48:55 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://sn.angry.im/)" 
19:48:55 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.sdf.org/)" 
19:48:55 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://mess.casa/)" 
19:48:55 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/1.6.1 +27 (bf7ce63); +https://is.a.qute.dog/)" 
19:48:55 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://bsd.network/)" 
19:48:55 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://toot.cafe/)" 
19:48:56 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://mess.casa/)" 
19:48:56 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://toot.cafe/)" 
19:48:56 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chat.cdstm.ch/)" 
19:48:56 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://chat.cdstm.ch/)" 
19:48:57 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://cmx.im/)" 
19:48:57 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://cmx.im/)" 
19:48:58 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://sdfn-01.ninjawedding.org/)" 
19:48:58 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodonten.de/)" 
19:48:58 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://pleasehug.me/)" 
19:48:58 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://sdfn-01.ninjawedding.org/)" 
19:48:58 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodonten.de/)" 
19:48:58 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://pleasehug.me/)" 
19:48:58 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://pleasehug.me/)" 
19:48:58 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:48:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.2.0rc2; +https://edge.mstdn.jp/)" 
19:48:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.1; +https://weirdgone.pro/)" 
19:48:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://social.nasqueron.org/)" 
19:48:59 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://social.nasqueron.org/)" 
19:48:59 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://coisland.xyz/)" 
19:49:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mstdn.beer/)" 
19:49:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.1; +https://weirdgone.pro/)" 
19:49:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.2.0rc2; +https://edge.mstdn.jp/)" 
19:49:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://dragon.style/)" 
19:49:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://coisland.xyz/)" 
19:49:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.1.3.-vcity; +https://vcity.network/)" 
19:49:00 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://sleeping.town/)" 
19:49:00 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.1.3.-vcity; +https://vcity.network/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mstdn.beer/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://weirder.earth/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://dragon.style/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://sleeping.town/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.1.2; +https://nethole.us/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://hostux.social/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://hostux.social/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://beepboop.one/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://weirder.earth/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://lgbtq.cool/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1-mek; +https://social.mecanis.me/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://beepboop.one/)" 
19:49:01 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://tenforward.social/)" 
19:49:01 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.0.0; +https://dragon.style/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://mastodon.hong.io/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.1.2; +https://nethole.us/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://lgbtq.cool/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1-mek; +https://social.mecanis.me/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mathstodon.xyz/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://mastodon.hong.io/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mstdn.beer/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mathstodon.xyz/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.1.3; +https://social.coop/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://scifi.fyi/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/2.2.2 (Mastodon/2.1.3; +https://seacow.social/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://nulled.red/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.1.3; +https://social.coop/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://nulled.red/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://tenforward.social/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://Mr.Lees.Greater.Hong.Kong.International/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.rocks/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.fr/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://anticapitalist.party/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.rocks/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.fr/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc1; +https://scifi.fyi/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.2.0; +https://Mr.Lees.Greater.Hong.Kong.International/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://monsterpit.net/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://anticapitalist.party/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://linuxrocks.online/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc3; +https://monsterpit.net/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://share.elouworld.org/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.jp/)" 
19:49:02 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://selfy.army/)" 
19:49:02 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://share.elouworld.org/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://selfy.army/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/2.2.2 (Mastodon/2.1.3; +https://seacow.social/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://ephemeral.glitch.social/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://haupt.bahnhof.cz/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mstdn.beer/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://haupt.bahnhof.cz/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://koyu.space/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://ieji.de/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://ieji.de/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://koyu.space/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc5; +https://geeks.one/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://linuxrocks.online/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc5; +https://geeks.one/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://ephemeral.glitch.social/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://fosstodon.org/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://soc.ialis.me/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc2; +https://icosahedron.website/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc3; +https://soc.ialis.me/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://fosstodon.org/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tcit.fr/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1rc2; +https://icosahedron.website/)" 
19:49:03 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.codingfield.com/)" 
19:49:03 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mstdn.jp/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://social.tcit.fr/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://mastodon.codingfield.com/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://cybr.es/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://cybr.es/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://campaign.openworlds.info/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://hex.bz/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://campaign.openworlds.info/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://writing.exchange/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://todon.nl/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://functional.cafe/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://tiny.tilde.website/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://functional.cafe/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://writing.exchange/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://todon.nl/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://slime.global/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://social.weho.st/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://framapiaf.org/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mamot.fr/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://tiny.tilde.website/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://framapiaf.org/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://hex.bz/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://slime.global/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://social.weho.st/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mamot.fr/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://screech.social/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.conxtor.com/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.3.3; +https://switter.at/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://screech.social/)" 
19:49:04 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.conxtor.com/)" 
19:49:04 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lgbt.io/)" 
19:49:05 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.3.3; +https://switter.at/)" 
19:49:05 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://lgbt.io/)" 
19:49:05 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://toot.cat/)" 
19:49:05 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.srv.hagen.coffee/)" 
19:49:05 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.eifelcoworking.space/)" 
19:49:05 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://mastodon.srv.hagen.coffee/)" 
19:49:05 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://mastodon.eifelcoworking.space/)" 
19:49:05 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:49:06 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://knzk.me/)" 
19:49:06 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.0.0 (Mastodon/2.3.3; +https://toot.cat/)" 
19:49:07 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.1; +https://knzk.me/)" 
19:49:07 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:49:09 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc5; +https://horsecr.app/)" 
19:49:09 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0rc5; +https://horsecr.app/)" 
19:49:09 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.3.0 (Mastodon/2.4.0; +https://pawoo.net/)" 
19:49:09 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:49:10 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15" 
19:49:10 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.3.0 (Mastodon/2.4.0; +https://pawoo.net/)" 
19:49:10 +0000] "GET /favicon.ico HTTP/1.1" 404 143 "https://test.ondrovo.com/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15" 
19:49:22 +0000] "HEAD /index.php HTTP/1.1" 200 0 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://eldritch.cafe/)" 
19:49:22 +0000] "GET /index.php HTTP/1.1" 200 76 "-" "http.rb/3.2.0 (Mastodon/2.4.0; +https://eldritch.cafe/)" 
19:49:34 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:49:59 +0000] "GET / HTTP/1.1" 200 76 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Safari/537.36 OPR/46.3.2246.127744" 
19:50:36 +0000] "GET / HTTP/1.1" 200 76 "https://chaos.social/web/accounts/21070" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36" 
19:50:36 +0000] "GET /favicon.ico HTTP/1.1" 404 200 "https://test.ondrovo.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36" 

It's may not be a huge deal now, but if the network grows and this keeps happening, it might be problematic. Consider when eg gargron boosts a toot - basically the entire fediverse will query that page within seconds to minutes (to make thumbnails nobody will even look at unless they open the toot in the detailed view).

[Gargron]

To those suggesting that the sending server provide the link preview:

• Can the sending server be trusted about that?
• Does the sending server know what we want from a link? Think Pleroma vs Mastodon vs Pixelfed vs Misskey. Think OpenGraph vs OEmbed vs some technology that doesn't exist yet but will in the future

[ashfurrow]

Can the sending server be trusted about that?

Yeah that's for sure been my concern. We've seen instances running malicious code modify payloads when federating before.

[MightyPork]

This may have been proposed before and isn't ideal, but how about using
lazy load for the thumbnail, requesting it on demand? I would imagine, for
the majority of statuses, the preview won't ever be used on the servers it
federates to.

On Tue, Jun 12, 2018, 11:09 PM Ash Furrow [e-mail] wrote:

Can the sending server be trusted about that?

Yeah that's for sure been my concern. We've seen instances running
malicious code modify payloads when federating before.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/4486#issuecomment-396734759,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB8lHppOEMXsxXB4likGHLv_KZNrgE70ks5t8C4MgaJpZM4OqUaH
.

[baby-gnu]

Maybe the preview could be delayed until someone select the toot for display?

This way, each server will query the URL only when one of it's own users display the toot, human does the randomised delaying.

[Tagadda]

Can we attach the preview to the toot as we attach custom emoji and media ?

This way, bots could even create embed view, or enriched toots. (as embed stuff on Discord)

[lfuelling]

Has anyone ever tried this out on a large scale?

I can't believe that the requests would be enough to cause a DoS.

[Gargron]

It's not quite a DoS but it can saturate a small web server that only has a few threads to work with. So it's a problem.

@MightyPork's suggestion is probably the most viable one.

[Deimos]

@Gargron should there be an issue left open for this to track that it's still something that will need a better solution eventually? The one-minute spread is a decent workaround at the current scale, but it's not really a solution overall. If there are 50,000 Mastodon servers someday, they can't collectively dump almost 1000 reqs/sec on every link that gets posted. Much lower levels of traffic than that take sites out regularly.

[nightpool]

right, but the operative number isn't "total number of servers" it's "number of followers the average person has" (because posts are only distributed to servers that have users that follow you). that number doesn't scale with growth, really, above a certain point. popular people on Twitter already take sites down by tweeting out links, mastodon is no different here.

also, by the time we regularly have mastodon users with followers across 50,000 unique servers (rough estimate, this means mastodon users with 500k-1m followers), you'd hope we'd be much better about caching being a standard part of web deployment.

[Deimos]

It's quite a bit different than Twitter. This behavior of Mastodon actively makes it worse, and can actually kill the site before any real people even click on the link. As people said above, pushing it off on the person running the server isn't really fair. Even sites like Hacker News regularly take out small/unprepared sites, and HN only sends about 10 req/sec. Mastodon can already exceed that level pretty easily.

[rhaamo]

The one-minute spread is a decent workaround at the current scale, but it's not really a solution overall.

Personally I don't consider it a decent workaround much.
Doesn't need to wait 50k servers to get a website down, just publish a public link and manage to get it boosted from mssoc.
Guaranteed to kill that website instantly, I'm still seeing small websites being killed that way.

[angristan]

I'm still seeing websites "killed" too

[Gargron]

I disagree with nightpool on this one and I am deeply uncomfortable with the effect linking a site on Mastodon has. That's not OK. Unfortunately what can we do here? We can try to spread requests over longer than a minute at the detriment of user experience (the previews are most needed when the post is at the top of your feed, after that the post itself quickly becomes irrelevant), but we can't remove the fetching itself. If only the origin server was responsible for fetching the preview, it would alleviate this problem, but its data could not be trusted by everyone else, and it might not even produce data in the right format at all if it's different software!

[ghedipunk]

Don't fetch a preview until a user wants one.

Change the UI so that previews start out collapsed or otherwise invisible.
Add some obvious element to let people know that a preview might be
available.

When the user clicks that element, attempt to fetch a preview and cache.

This way, servers where the users never fetch the preview will save (very
slightly) on resources, and since users take significantly longer than a
minute to view posts in many cases, we're further spreading out the load.

On Wed, Jun 12, 2019 at 7:16 AM Eugen Rochko [e-mail]
wrote:

I disagree with nightpool on this one and I am deeply uncomfortable with
the effect linking a site on Mastodon has. That's not OK. Unfortunately
what can we do here? We can try to spread requests over longer than a
minute at the detriment of user experience (the previews are most needed
when the post is at the top of your feed, after that the post itself
quickly becomes irrelevant), but we can't remove the fetching itself. If
only the origin server was responsible for fetching the preview, it would
alleviate this problem, but its data could not be trusted by everyone else,
and it might not even produce data in the right format at all if it's
different software!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/4486?email_source=notifications&email_token=AAB46PSI34L3JTU4YPYDESDP2EALLA5CNFSM4DVJI2D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXQSEJA#issuecomment-501293604,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB46PVW5H2634TCZVYBOJLP2EALLANCNFSM4DVJI2DQ
.

[Gargron]

Change the UI so that previews start out collapsed or otherwise invisible

That defeats the whole point of having previews though. Otherwise people would just click on the links. That's not a solution.

[lfuelling]

I wonder if any site was truly killed by a Mastodon post.

Hiccups and maybe an outage (for others) while the instances do all their requests, but killed?
As in: "The server process exited and didn't restart"?

I can't believe that. Feel free to prove me wrong.

[rhaamo]

sorry to not use The Right Word©, I'm here to discuss the issue, which is "mastodon can be used as a DDOS tool".

and no, websites are just temporarily down, and not "OMG THEY ARE KILLED", but that doesn't change the issue at all.

thanks for that useless comment.

[lfuelling]

I consider a DDOS attack to be more (and longer) than a few seconds of lots of requests...

Those are (imo) not DDOS attacks:

• https://github.com/tootsuite/mastodon/issues/4486#issuecomment-319505423
• https://github.com/tootsuite/mastodon/issues/4486#issuecomment-395076695

Talking about useless comments...

[coreyreichle]

This isn't really a DDoS, per se. More a result of popularity. The proper term is "slashdotting", afaik.

That being said, something probably should be done to prevent this, and the origin-preview is probably the best way.

Yes, the origin can craft malicious images. But, then again, that's the case today as well.

[rhaamo]

Yeah maybe not a "real" DDoS but pretty much anyone including @Gargron here acked the issue resulting in a denial of service (personally making a website down for 15+minutes is making it out of service, even if it's temporarily).

As said by @ghedipunk fetching/generating that preview on first view would be the best option as it might spread out the load better than 1min.

[Deimos]

As a specific example, someone tooted a link to my site early this morning. This is what that area of my traffic graph looks like, for requests that make it through to the actual Pyramid app (i.e. not static files). The baseline traffic at the bottom is about 1 req/sec:

That's 740 servers, sending almost 1500 requests in one minute that actually reached the app (each server sends both a HEAD and a GET), and would be over 2200 if you also included all the additional requests for the image to include in the embed.

I obviously personally understand the source, but this could seem like a really worrisome event to someone that doesn't know what Mastodon is. Looking in their logs to figure out what knocked their site offline and seeing thousands of lines that include http.rb/3.3.0 (Mastodon/2.9.0; +https://some.mastodon.url/) isn't going to be a good first impression. I wouldn't be surprised if some sysadmins end up just reflexively blocking the user-agent since it appears to be strange, automated, DDoS-like traffic.

The pedantry about whether this is a "real DDoS" and so on doesn't matter at all. It's not a good behavior, @Gargron recognizes that, and I hope some work goes into finding a solution. I'm not familiar enough with the internals of Mastodon to know what that solution might be, but it can't be every individual server always fetching directly from the source.

Maybe a possibility is something like "trusted scraping servers" that can scrape and federate the results, or maybe link-previewing is just a feature that's fundamentally incompatible with Mastodon. It's a different model, and features that are possible on a centralized service can't necessarily always work on a federated one too.

[ksyufi]

This is happening to my website. I have a rather small webcomic website hosted on a potato, and I noticed that it's being knocked over pretty regularly. My logs are full of GET requests from different Mastodon instances, like http.rb/3.3.0 (Mastodon/2.9.3; +https://some.url/).

The link to my site was in my profile fields and I had the html tag for verification. I do post links to it whenever I update my comic, but I usually attach a picture, so website preview isn't even needed. But today the server was down for an hour after I tooted something not related to my website at all.

What can I do to avoid this issue? For now I've removed the link from my profile metadata fields.

[Gargron]

I guess we could try to skip the HEAD request and just do a GET straight away? That would halve all requests. Beyond that, I suppose we could increase the maximum delay to spread out the requests over longer than a minute...

[marrus-sh]

honestly i think the concept of “trusted servers” needs to be explored more, as increasing the delay is just postponing the problem. obviously one cannot simply trust card information from any old server. but i could see a feature like with trending hashtags, where after users of instance A follow enough users of instance B (say, 10?), admins of instance A are prompted to either “trust” or “not trust” the instance.

this would prevent suspicious instances from conveying false information without duplicating work across instances that regularly talk to each other and have a level of already­‑implicit trust

[bortzmeyer]

What can I do to avoid this issue?

Remember that, if Mastodon can do it, anyone can, even with a small botnet (there are not so many Mastodon instances). So, it mean that your site is not up to the task of an Internet public site. I suggest that you explore ways to make it more robust: static instead of dynamic (or static made from a dynamic site with httrack), use of a cache like Varnish, etc.

[feld]

This has been on my mind for a while too.

Pleroma does not immediately fetch the preview upon federation, but caches the OpenGraph/OEmbed/Twittercard data after first load of the post. A webserver should only see two requests per Pleroma server (preview metadata, then the image which is loaded into hopefully-enabled MediaProxy cache).

It's still not ideal, but I don't see a sane way of solving this without centralization and honestly if your site can't handle this it can't handle real internet traffic i.e., the Slashdot effect. It's a hard lesson to learn but we just need our sites to be running on more efficient tech stacks.

Thanks for listening to my TED Talk.

[jlaufersweiler]

Let's not blame the victim for not having robust caching and DoS hardening
on his indy comic site that presumably holds up just fine under their
normal readership load when an unintended consequence of Mastodon's
architecture suddenly overwhelms his site. Many small sites haven't gone
through those sorts of growing pains yet, especially ones run by creators
who are learning as they go, Their fragility doesn't absolve other
operators from the responsibility to avoid being a bull in a china shop.

On Wed, Oct 9, 2019 at 11:57 AM Mark Felder [e-mail]
wrote:

This has been on my mind for a while too.

Pleroma does not immediately fetch the preview upon federation, but caches
the OpenGraph/OEmbed/Twittercard data after first load of the post. A
webserver should only see two requests per Pleroma server (preview
metadata, then the image which is loaded into hopefully-enabled MediaProxy
cache).

It's still not ideal, but I don't see a sane way of solving this without
centralization and honestly if your site can't handle this it can't handle
real internet traffic i.e., the Slashdot effect. It's a hard lesson to
learn but we just need our sites to be running on more efficient tech
stacks.

Thanks for listening to my TED Talk.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/4486?email_source=notifications&email_token=ABWNFAV56VBKW7W2KIG74M3QNYEPLA5CNFSM4DVJI2D2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAYSGDQ#issuecomment-540091150,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWNFAS7VVVZ5XF3IKAVW4DQNYEPLANCNFSM4DVJI2DQ
.

[feld]

Let's not blame the victim for not having robust caching and DoS hardening on his indy comic site that presumably holds up just fine under their normal readership load when an unintended consequence of Mastodon's architecture suddenly overwhelms his site. Many small sites haven't gone through those sorts of growing pains yet, especially ones run by creators who are learning as they go, Their fragility doesn't absolve other operators from the responsibility to avoid being a bull in a china shop.
…

You're being disingenuous. You don't need a CDN or "DoS hardening" to handle a thousand requests per second.

[Gargron]

I've already changed Mastodon's code to perform one GET instead of HEAD+GET, which should half the amount of requests any link gets from the fediverse with 3.0.

As I said it's really not alright to hammer websites like that, yes it's not an unusually high amount of traffic for the internet but that's why "slashdotting" is a verb, many websites can't handle something like that, and I do not think discussing that part further is constructive.

If we fetched the preview when a status was viewed instead of when it arrives (plus randomized delay) as we do now, it would introduce its own set of problems, since performing synchronous HTTP requests within a request/response cycle would (okay, ironically) impact Mastodon's performance, and if at least one person on each server would be watching the federated timeline you would be exactly where we are now, but without even a randomized delay to spread the traffic out.

[sorz]

Remember that, if Mastodon can do it, anyone can, even with a small botnet.

It's also my concerning. Normal use cases will make (only?) thousands of requests. But for malicious users, it's possible an amplification attack that turn one POST request to ~1000x GET requests.

Do we have any rate-limiting for this? For example, in each instance, at most X requests within Y minutes for any particular domain will be made.

[Gargron]

No, we don't have per-domain rate limits for link fetching. I don't see a one-size-fits-all rate limit given that e.g. YouTube links probably get posted the most and YouTube can handle the preview fetching just fine, while a small site would need a very low rate limit. Also I guess one of the problems with that suggestion is that the root of the problem stems from different Mastodon servers acting in (near) unison, if each implemented rate limits, the same amount of servers would still be hitting the same domains the same amount of times.

[ddevault]

"Hey guys, you know that your software just casually DDoSes random internet hosts all the time, right?"

Cue 50 comments of blame shifting over 3 years and no one actually solving the problem

[Gargron]

OK @ddevault, what do you want to do:

1. Remove link previews
2. Allow origin servers to tamper with link previews (incl. HTML) and standartize their representation in JSON-LD
3. Move link preview fetching to a centralized service and operate said service
4. Hardcode domains from which link previews can be fetched to high-profile proprietary services (e.g. YouTube, Vimeo, Twitch, etc)
5. Spread the randomized fetching delay even further, making previews not show up for a larger number of users who are looking at the post

Or maybe, a couple thousand GET requests spread over a minute isn't that unusual on the web and is the best trade-off considering the alternatives. And remember, that's the worst-case, when the link has never been posted before and none of the servers have a preview already.

Also, it's not true that nothing has been done over the course of this GitHub issue. Literally at the start of the issue I added the randomized delay over the span of a minute. Also, what initially used to be a HEAD request plus a GET request has been reduced to just a GET request, that literally halved the amount of requests made.

So do you have a solution or are you just contributing to the exact situation you are describing?

[ddevault]

If it were up to me, I would probably not fetch previews at all until the user clicks the post and asks for them.

What I definitely would not do is throw up my arms and say "oh well, it can't be helped, I guess we'll just DDoS everyone!"

If you can't find a solution, remove the feature. Don't ship things which freaking DDoS the internet.

[neekz0r]

Or maybe, a couple thousand GET requests spread over a minute isn't that unusual on the web

It's only usual for DDoS. Not something a good netizen/software should be doing. This is very short sighted thinking. As I would think everyone here would want Mastadon to replace twitter -- at least that is the hope and dream.

If it does get to be that size, this problem will only be exacerbated and go from "a few thousand" to "a few million".

Thus, the current behavior is not a long term sustainable "feature".

[ProgVal]

An alternative: relying on Signed HTTP Exchanges (not to be confused with HTTP Signatures)

I don't like this spec very much because it's used by Google to push AMP, but it has a core feature that can be used here: original servers sign the content of the resource being fetched. Then Mastodon can re-serve this content with the signature.

That way, servers which offer the signature immediately get a decrease in traffic if the initial Mastodon instance is well-behaving.

If the initial instance is malicious and strips the signature, then I'm not sure there is a solution. Receiving instances could check against their cache to see if that server supports signatures, but that's not reliable. Signed HTTP Exchanges is still a draft; maybe it could be amended to allow servers to indicate they will sign all requests?

[bortzmeyer]

If you can't find a solution, remove the feature. Don't ship things which freaking DDoS the internet.

Can we keep a cool head? If 2000 GET requests were to "dDoS the Internet", the Internet would be in serious danger! Even a very small botnet could do the same.

[ddevault]

Ah, yes, blame the victim.

Mastodon causes other websites to go down. That's the facts, plain and simple. Not every website is designed to cope with load, and that's fine.

[bortzmeyer]

4. Hardcode domains from which link previews can be fetched to high-profile proprietary services (e.g. YouTube, Vimeo, Twitch, etc)

You certainly know it but, for the benefit of the people who don't, just a reminder that you do not need to be YouTube or Cloudflare to handle the 2000 Mastodon instances. My own Web site is a a small VPS at a low-cost hoster and has no problem when I post a link on Mastodon. The problem comes from sites where you need 100000 lines of PHP and 5000 of SQL to display one page.

[ddevault]

It's not the website's fault that they were DDoS'ed. I'm sick of this entitled opinion that just because it's a BeSt PRaCtICE that you should be able to take hundreds of requests per second, it's suddenly okay to DoS people.

Knock it off!

[bortzmeyer]

it's suddenly okay to DoS people.

It's not, but a DoS is when you deliberately attack people. Again, I suggest that the problem won't be solved by name-calling, specially when it's wrong.

[ddevault]

What makes you think a DoS has to be deliberate?

And now that you know about it, continuing to do it is deliberate.

[Kleidukos]

Now now, children, let's keep our heads cool.

Drew, I know that shouting at entitled people is your brand, but all you manage to do is making yourself ridiculous when challenging Stéphane Bortzmeyer on his knowledge of the nature of Denials of Service.

Having lots of requests to actors of the federation is inherent to its nature and was predictable. This is the price to pay to be connected to a lot of people without the proxy of a centralised service.
Glancing at the request numbers that have been reported in this thread, they do match with what I would expect from the number of participants involved.

That being said, while the maintainers of AP software can (and should) avoid relying on building blocks that are unsuitable for that kind of traffic, I will also claim that it is the duty of those in charge to protect the infrastructure of those who do not willingly participate in the orgy of HTTP requests that happens each time someone posts a status with a URL pointing outside of the federation.

Pleroma's MediaProxy and other tactics belonging to the UI (like lazy loading the meta-data until it's actually wanted for display) are strategies that should see wider adoption. Maybe the AP relays could be used to provide a buffer? Their purpose, after all, is to handle traffic.

[feld]

This problem will only get worse as the fediverse grows. I think that the rich previews are a crucial feature and should not be removed. It's not possible to "rate limit" it, because we don't control all nodes in the federation.

Perhaps we should make the original federating server fetch before publishing and include it in the activity as additional metadata so it's only pulled once. Images could be base64 encoded.

We already trust the content of the activity, so is it so much worse to do this? Someone will definitely think it's amusing to craft bogus rich previews and send them around the fediverse, but there's already much worse things they could be doing.

[nightpool]

I'm not against trusting the title/description/image from federated servers (Tumblr even allows users posting Link posts to "customize" the title and description in the embedded display, so there's clear precedent for this being a user-controllable field), but there are other factors here that are more sensitive. If the user sees a rich-media embed (OEmbed or OpenGraph embed) with a domain of youtube.com, and they click into it, they expect their data to be sent to youtube.com. If we trust the OEmbed endpoints or OpenGraph embeds sent to us by remote servers, that allows remote servers to masquerade as potentially user-trusted domains and harvest IP addresses, correlate cookie values, etc, even with our current click-to-play embed model.

[feld]

If we trust the OEmbed endpoints or OpenGraph embeds sent to us by remote servers, that allows remote servers to masquerade as potentially user-trusted domains and harvest IP addresses

How? What I'm proposing is:

1. User makes post with link
2. User's server generates the link preview metadata by scraping the OEmbed/OG
3. User's server includes this data in the payload during federation and included in the response to any server asking for the activity (it's literally embedded in the activity)

This does not create any scenario where an end user's IP address can be exposed to any server in the fediverse except the one they login to.

[ProgVal]

@feld OEmbed is basically an iframe to whatever link is in the original toot. The user's server doesn't rehost it (imagine if it was a 10h youtube video...)

[feld]

@feld OEmbed is basically an iframe to whatever link is in the original toot. The user's server doesn't rehost it (imagine if it was a 10h youtube video...)

Ignore OEmbed, I don't really care about that. It was in my original suggestion because my brain isn't working on a Friday afternoon. OEmbed is out of scope because it breaks CSP rules anyway, so nobody should be using it if they care about security. It should only be allowed to certain trusted endpoints, none of which are going to be a low resource server from a indy creator.

All I care about here are TwitterCard/OpenGraph metadata: the link title, description, and an image we can b64 encode. The scraping for those meta tags are what's harming people, not OEmbed.