Mastodon: [Discussion] Switching away from Postgres for horizontal scaling

[yookoala]

Probably related to #591 but not exactly.

Mastodon stack is using Postgres as the database. And as we all know, Postgres is scaled vertically. With the speed we aggregate data (especially with high usage instance like pawoo.net), and with the current design where federated messages are stored in the subscriber's instance, we'd probably run in to scaling issue.

The question is that "should we switch to another database that does horizontal scaling?"

In particular, I read about CockroachDB and it is a horizontal scaling database. It basically support Postgres protocol and many of the data type. It could be a drop in replacement for database (depends on what feature we're using in Mastodon currently).

We already have discussion like #1847. And we'd probably have no real problem in a very long time. So no rush.

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [ ] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[expenses]

I think you'd need to chat with @Gargron about scaling, who wrote a blog piece about it.

How hard would it be to implement the ability to change the database backend?

[kstrauser]

And as we all know, Postgres is scaled vertically.

That's not remotely true. It's very possible - I've done it - to set up an arrangement where one node accepts writes and then propagates them to N read-only replicas.

PostgreSQL is bored with less than 10,000 inserts per second. Even pawoo's traffic is teensy in comparison to what a single PostgreSQL instance is capable of writing.

[yookoala]

@expenses Thanks! I have read that. But I'm not sure this is the right solution to max_connections issue in PostgreSQL.

[yookoala]

@kstrauser Whether Postgres can withstand a hight inserts-per-second or not is not relevant to the scaling model of the software.

But it is still nice to know that Postgres can hold up well with the stress in pawoo's traffic.

[kstrauser]

@yookoala Well, it kind of is, isn't it? If you can handle a heavy write load and can fan out the reads to a large number of consistent replicas, you have a great scaling story.

I've never built web services that handled more than a few hundred thousand requests per second, but I think even that should be sufficient for any single Mastodon instance for quite a while to come.

[yookoala]

@kstrauser You point being "Postgres can handle the scale of Mastodon instance in near future" does not conflict with my point "Postgres is scaled vertically, and there are horizontally scalable database in Porstgres protocol".

As I stated on top, I don't think this is solving a problem we currently, or in a near future, have. This is merely a discussion I want to raise. I don't even have a conclusion on this yet.

[kstrauser]

@yookoala I guess my concern is the original question implies that large PostgreSQL installations scale vertically, when there are many ways to scale its clusters horizontally. Don't get me wrong - it's fun to imagine these things as thought experiments! But there's a very real cost in prematurely optimizing a part of the Mastodon stack that's very unlikely to be a bottleneck. We just don't know what its failure mode might look like and therefore don't have any requirements for what a replacement would need to do.

I think we'll much sooner hit the point where it'd be nice to route read and write requests separately. Ideally, Mastodon would allow at least two database configurations: one for inserts/updates and another for selects. You can get the same functionality with pgbouncer-rr. But if Mastodon _knows_ that a query is select, it'd be faster for it to send that to a (random) read replica than to let middleware inspect it to see where to send it.

[siuying]

For system like Twitter or Facebook, a different kind of database is certainly required. But for the problem Mastodon solving, I'd see the bottleneck is not posgres, and replace the database would not solve any problem we have.

[wxcafe]

This is way too soon to have this discussion and there are lots of other things to do before that, without even talking about the fact that postgres is not the bottleneck and isn't gonna be for a long while. I'm closing this issue.