Mastodon: Multi-tenancy with custom domains

[Gargron]

More technical look at #897

Idea: Let users point DNS records to an existing instance, configure specific user to be served as-if from that domain.

What parts must work correctly for this:

• [ ] Separate field for storing whether an account is remote or local rather than relying on domain being null - store custom domain in that column?
• [ ] All functions that check whether an account is local must check the custom domain as well as the "local domain" of the instance
• [ ] Webfinger must answer queries for the faux-remote account
• [ ] Other accounts must not be served from that custom domain. Accessing the website through the custom domain should be limited to only public content from the owner of the domain.

Concerns:

• Switching instances while keeping the same domain will present the same problem as erasing/reinstalling an old instance on the same domain - other servers will remember data that you don't, any numeric IDs may reset and overlap (but generated string URIs include not just numeric ID, but also timestamp of creation, so that's not a big concern)
• Might be the first credible case for exporting/importing data verbatim, e.g. including all small details. Just like with local users having a custom domain, statuses should be amended to be local and able to have verbatim URIs like remote ones

[ZiiX]

Great option to consider! You know I'm for it. With this feature, I'm thinking it would be also awesome to consider personal file/media settings as an option with this. WebDAV, S3, Dropbox, Amazon, Google, etc - this allows their media hierarchy to follow them too.

[ramsey]

I feel like there are a lot of security/identity concerns to consider, especially with importing data verbatim. I could forge an import and pretend to be someone else, replaying all their posts into the new server. There’s no mechanism for signing posts and asserting one’s identity.

[saper]

@Gargron what about having a separate domain table and foreign key to that.

Also I think the primary domain should be treated just like all those tenants. Do you have a reason to have it "special"?

[Gargron]

@saper Initial design decision was:

• Your domain could be adjusted (even just only at the very start when you're setting up/troubleshooting)
• Don't want to have 20.000 copies of the same string in the db if it can be avoided by making it null and generating local URIs on the fly
• Simple distinction for local vs remote content - remote content has a non-null URI

Of course right now we're exploring a use case where, perhaps, that design should be replaced... At the same time, you're right, we could have a separate domain table, then there wouldn't be 20.000 copies of the same string. Though you'd need to join the accounts and domains table on basically every single query.

[saper]

Though you'd need to join the accounts and domains table on basically every single query.

This is something PostgreSQL should not have any problem with. Definitely we should not be storing
multiple copies.

[jeremyBanks]

Will it be possible to point to a different user name? If you need to use the same name across instances, that could limit your ability to migrate.

[ThibG]

Off the top of my head:

• Some of this is already handled with the distinction between LOCAL_DOMAIN and WEB_DOMAIN, except that you'd need to use a custom user-specific domain if not null instead of LOCAL_DOMAIN, which is easy when generating something, but can get messy when looking up an user. Also, Mastodon still makes some assumptions that are broken when WEB_DOMAIN is different from the user's domain (see #2012 and #2032), although until now, this can be worked around, but…
• What about username conflicts? Should user@redirected and user@local conflict or not? If not, the workaround for misdirected webfinger queries (https://github.com/y0t4/mastodon/commit/46555ce14df83391cdf222e574d170a7eebe7a8c) can't work in this setting, as it would need to disambiguate.
• Further, should the URIs of redirected accounts follow the same template as local ones? If so, and user@redirected and user@local are different users, how to disambiguate? The easiest way to handle that is probably to have https://web_domain/@user@redirected and https://web_domain/users/user@redirected etc.
• With that in place, using a remote Mastodon instance for your domain seems easy: just redirect (or proxy, but #2375) /.well-known/host-meta (and possibly /.well-known/webfinger), or use a CNAME RR for your domain, if it's to be only used for Mastodon.
• Migration probably will be a bit tricky: in OStatus, users are not identified by their acct: (user@redirected) URIs but by their ActivityStream actor URI (which is so far an URL to the user's web profile, and will thus reference the “backend” Mastodon instance). The easiest way to migrate seem to just point to another backend, however, your actor URI will most probably change as a result. I'm not sure how bad a thing that is. Also, it won't be immediate, as remote instances having your account details will need to refresh it from webfinger to switch to the new instance, and that's currently only done in a few situations, and at most once a day.

[semarie]

having this kind of configuration at DNS level would be great. But isn't more simple to have it at webserver level ? it would mean having a minimal code on remote webserver side (could be done with simple PHP script and/or webserver redirection).

As example, considere [email protected] who want to use [email protected]

On the remote side (my.domain):

• /.well-known/host_meta : reply with LRDD link to webfinger on my.domain
• /.well-known/webfinger : get information (mostly magic-public-key) from mastodon.social, and build a reply with acct:[email protected]. Use local URL for atom, salmon, ...
• for URL entries points (atom, salmon, ...), use HTTP redirection (Location HTTP Header) to point to mastodon.social

On mastodon side (mastodon.social):

• permit the user to config is account as [email protected]
• use acct:[email protected] in replies

[russss]

I'm keen on @semarie's proposal, and in fact I started looking into the feasibility of it a while ago. The advantages as I see it are:

• You don't need a separate hostname purely for this - which means (for example) my username could be the same as my email address.
• Multiple users on the same domain could be housed on different Mastodon instances.

This would be similar to how OpenID works, and it feels like a more "federated" situation than pointing domain names around.

It also seems that in the original proposal, the host Mastodon instance would have to obtain an SSL certificate for each domain (or serve them unsecured which is annoying). Even with LetsEncrypt this is fiddly.

[saper]

here is what I understand under domain multi-tenancy:

Ability to point multiple A and AAAA DNS entries, add a new domain to the certificate and make sure the domains do not get mixed up.

1. Different domain names are treated like independent instances.

1a the simplest implementation would be just to reuse the code base and point to different databases on request. This is probably achievable now with a web server configuration. I think this should be tested prior to proceeding further (how to talk to redis, store media for multiple instances, mail support etc.).

Every instance is completely independent, has its own member namespace and they federate independently. They only get upgraded at the same time. You have to run db:migrate multiple times, with a new RUBY_ENV for ever domain in the loop.

I think there must be some folk trying this right now?

MediaWiki has an interesting hack that allows sharing only some database tables between such instances, not sure it would be helpful here.

A dedicated asset domain could be another thing, but it should be made in a way that does not break the possibility to export the domain to a fully standalone instance.

1b accept multiple WEB_DOMAINS and issues and hacks related to that. This is more tricky in the implementation as it might require a new domain table, new queries etc. This provides little more benefit to 1a but it carries more of the development effort. The web server/PostgreSQL configuration is simplified compared to (1a)and every domain lives independently in the webspace, with its own member namespace.

Interesting questions appear here, like should those instances automatically be federated or not?
(I think not, they should feel independent).

There is a possibility to shortcut the traffic between the domains internally for performance reasons (maybe not necessary).

1c Accept multiple domains under one certificate - basically it is the scheme used by Google multi-tenant Apps. Everything is under google.com/a/domainname/restoftheurl

The problem here is the initial redirect - how to securely redirect domain.fluffy to mastodon.social/a/domain.fluffy on the web? In case of SMTP it works because nobody validates SMTP domains in certs and we have MX records.

One solution could be to have some custom SRV record that is universally accepted (haven't checked the spec yet). StatusNet, bot frameworks etc should also use it.

A quick check with the WebFinger RFC tells me it only supports HTTP redirect to a different domain and not a well known URL but you still need to be able to accept initial request securely.

And here comes another variant:

2a. Some fluffy form of "user@domain" as an identity alias on mastodon.social without DNS entries. I don't know how would that possibly work, except for the user being [email protected]vall the time on the protocol level and just bring presented as @[email protected] on mastodon.social only.

2b. A variant of (2a) could be considered where mastodon.social tries to convince other instances that mrserious wants to be presented as @[email protected], which opens a whole can of worms (misrepresentation).

May be this use case should be further elaborated (maybe it is a separate issue).

I think there is a natural development progression to go from 1a via 1b maybe to 1c.

it would be good to refer to one of those variants as we speak (or define additional ones)

[s-leroux]

Great idea. But let me rephrase to see if I understand the typical use case:

Someone connects to me as @[email protected]

First we query the yesik.it DNS for the _mastodon TXT record.
If not found, "yesik.it" is assumed to run its own Mastodon instance.
Otherwise, the TXT record (CNAME?) holds my current instance. Say its "mastodon.social"

So, on the background, Mastodon will understand the _physical_ account name is @[email protected]
But in the UI and for all user interactions, only my _logical_ account name is used @[email protected]

Basically we add here a new layer of indirection. Or did I missed the point?

---

Several remarks:

1. It doesn't help if someone bookmarked my account URL (https://mastodon.social/@YesIKnowIT). Or does it?
2. What if several members of the _same_ domain have accounts on _different_ instances?
3. What if I actually made _several_ domains point toward the _same_ account (aka, @[email protected], @[email protected] ... being different alias to the same account -- just like I can do it for my email).

[saper]

@s-leroux so what you mean is something like federated alias.

Basically one more assumption had to be broken - not all users from @domain are hosted on a single instance.

That is another piece of complexity, basically "domain" server would need to provide redirects/pointers,
but I understand that the trouble is for the target host to accept user@otherdomain stuff and somehow treat it's as own

Not sure it's multi-tenant domain handling, though

[alexgleason]

I like this idea a lot, but not sure the benefits outweigh the costs.

I think the goal is to give users more control. The easiest way to do that now is to run your own instance. That's hard though for non-tech users, while setting some DNS values is easy given a tutorial.

Would it be better to make it easy to start your own instance? I feel like a 1-click install on a web host would be ideal. I think IndieHosters is supporting Mastodon, maybe they'd be willing to set something up like this. @pierreozoux

Basically I'm wondering what the goals actually are and how to best reach those goals. I can definitely see the benefits of supporting remote domain names, but running your own instance is more beneficial: you own/control your data, there's no reason to migrate, and it produces no technical debt to the project itself. The paradigm currently in place makes sense: if you're part of a server, you're part of that community and you take their name. If you're remote, you have a remote name. This is intuitive.

On the other hand, email has some similar paradigms and any service can send emails from your domain name with the proper authentication (eg Postmark). But this is usually a generic service and to my knowledge is not built into any email providers themselves.

[pierreozoux]

I think it is a tricky discussion.

In one hand, with one postfix, I can host various domains, and it is convenient.
As a user, I can migrate from one hoster to another, it is a bit painful, but it is doable.

On the other hand, on the web, one domain name means the control over your own piece of land, your rules, your design, so it also makes sense to have your own instance (from the indieweb principles).

It is not that hard to host a mastodon instance, or pay somebody to do it. And then, it is easy to just transfer from one server to another, or from one hoster to another. And the change that has to be made to add this feature are quiet heavy I guess. Then, they also have to be maintained.

I think that multitenancy, is always a headache for developers, and it is never really well done, so even if I have the option, as a hoster, I never choose that. It is always easier to migrate people with their own instances.
(All the options I hear, DNS, running a small app server will touch 0.01% of our user base IMHO)

I'm probably not the best judge to help here, but I think the discussion has to evolve around cost/benefit for the software in itself. I'm a fan of unix philosophy, and I think the effort can serve a lot better in other areas :)

And to conclude, I also think that the discussion should happen at the standard itsefl, at ostatus level.

[jeremyBanks]

"It's not hard to host a Mastodon instance", said one engineer to the next.

Dozens of projects have told themselves this, but the reality of self-hosting applications like this is that it does not happen for a non-trivial number of users, despite great efforts like Sandstorm. One-click install for end-users is not a responsible solution, unless you also have automatic fast and secure updating on multiple levels. We might get there one day, but don't let that ideal get in the way of a realistic solution.

Configuring DNS entries as a one-time (or at least rare) action is something that a moderately tech-competent person can handle. It will at least get Mastodon on par with email in terms of identifier portability, and is probably the most viable mid-term solution (assuming the current identifier scheme is preserved instead of moving to something cryptographic).

[DanielHeath]

Postfix is a good example of something that is easy to operate.
Enable unattended-upgrades and security patches happen magically.
You can automatically transfer all your mail from one provider to another via standard APIs.
This is still not enough - the vast majority of users prefer free-with-ads (eg gmail) or pay-for-hosting (eg fastmail).

Hosting your own Mastodon instance means caring about:

• Backups (you presumably care about your own history)
• Patching security bugs quickly (Much as I love rails, it requires more frequent patches than other stuff I run)
• SSL (letsencrypt has made this easier, but it's still a job to setup)
• Diagnosing & fixing issues (if something goes wrong, can you pull the relevant bit of the logs to file a bug?)
• Uptime (ok, it's a personal blog, but it still needs to be fixed when it crashes)

I build and operate production services for work - I don't need or want another thing to carry pager for.

[DanielHeath]

@s-leroux I think a mastodon-specific TXT record is unnecessary; a regular old CNAME should do fine.

I think that separate-but-related is the ability to redirect an account from an old name to a new name. It's nice to be able to change your identity (even on the same domain) without breaking all the old URLs and losing your federated followers - there's no reason this sort of redirect shouldn't work across domains too.

[s-leroux]

@DanielHeath The problem with CNAME is cannot coexist with _any other record_ for the same resource (except DNSSEC-related records).

In particular, you cannot have both a CNAME and a MX record for the same domain.
Practically, if you use a CNAME to point your bare domain to some "mastodon provider", you no longer are able to use a mail provider on your bare domain. The workaround being to create a sub-domain per service. But that feels ugly to me.

[DanielHeath]

You can use an A record with an MX record.

That gives you mastodon and email, but you'll have to use other domains for any other protocols.

If you need something else, you can run nginx as a reverse proxy, or host your own.

Thanks,
Daniel Heath

On 30 Jun 2017, at 6:42 pm, Sylvain Leroux [e-mail] wrote:

@DanielHeath The problem with CNAME is cannot coexist with any other record for the same resource (except DNSSEC-related records).

In particular, you cannot have both a CNAME and a MX record for the same domain.
Practically, if you use a CNAME to point your bare domain to some "mastodon provider", you no longer are able to use a mail provider on your bare domain. The workaround being to create a sub-domain per service. But that feels ugly to me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[s-leroux]

@DanielHeath I understand your point of view concerning someone having full access to his server.

But imagine a business or other kind of organization deciding to provide Mastodon access to their team.

However, they already have mail and web site on their primary naked domain (MX and A records). In most cases those services are externalized. The web server having chances of being a mutualized one.
The only standard record that _might_ "bind" their domain to some mastodon provider would be the SRV record. Unless we force the user to create a specific sub-domain? Anyway, in the cases I have in mind, if I explain to them they will now have to host themselves a server just to install a reverse proxy or some other service -- surely they will answer, "never mind, we will continue using TweetDeck".

My point is if we want to target a wide audience, any viable solution should satisfy those two constraints:

1. It should be possible to co-host Mastodon as well as classical email and web services on the primary naked domain
2. For the purpose of adding the "Mastodon alias", we should assume the user only has access to his domain DNS configuration.

[DanielHeath]

My issue there is that it breaks ostatus protocol compatibility to achieve something that can be implemented without breaking compatibility.

If the project didn't need to adhere to existing federation standards there are lots of nice features that could be added; I don't think that's what mastodon is about though.

Thanks,
Daniel Heath

On 30 Jun 2017, at 9:14 pm, Sylvain Leroux [e-mail] wrote:

@DanielHeath I understand your point of view concerning someone having full access to his server.

But imagine a business or other kind of organization deciding to provide Mastodon access to their team.

However, they already have mail and web site on their primary naked domain (MX and A records). In most cases those services are externalized. The web server having chances of being a mutualized one.
The only standard record that might "bind" their domain to some mastodon provider would be the SRV record. Unless we force the user to create a specific sub-domain? Anyway, in the cases I have in mind, if I explain to them they will now have to host themselves a server just to install a reverse proxy or some any other service -- surely they will answer, "never mind, we will continue using TweetDeck".

My point is if we want to target a wide audience, any viable solution should satisfy those two constraints:

It should be possible to co-host Mastodon as well as classical email and web services on the primary naked domain
For the purpose of adding the "Mastodon alias", we should assume the user only has access to his domain DNS configuration.
Re-reading the entire discussion, I cannot resist quoting @saper :

Not sure it's multi-tenant domain handling, though

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[duck57]

One other use case for this (assuming I understand correctly) is for a current self-hosting user to add other domains to their existing instance. This probably will mostly be used for novelty accounts, but having the ability to use one instance of Mastodon on your server to serve several domains would be a nice-to-have feature and there are probably better uses than silly account domain names that I'm just not thinking of at the moment.

[azbulutlu]

some questions, as a user.

1) I use shared hosting, I have my domain, that has my website and email address. Can I use this feature? (not everyone can afford to pay for additional vps hosting. not everyone has time/skill/energy to run a server. even if a person has a vps, running a public facing server with good uptime is a lot of overhead on top of that)

2) What is my profile url? (note that when giving social media links on a website, I use the profile url).

3) what is my mastodon handle? (if I give someone my profile link, ideally they should be able to add me back, like they do with the profile url of a regular account. So follow button, or typing @username@customdomain or pasting the link and finding the account that way)

4) (potentially silly question) in terms of the idea for shared certificate, my domain already has a certificate, does that mean I cannot use this domain?

what are the answers in case I use a subdomain that is dedicated to mastodon redirecting instead (e.g. social.mycustomdomain.com)

@duck57 A scenario would be that: you can have 2 domains (e.g. both your personal page and a project/blog) You might want to have "official" accounts for both, without having 2 mastodon installs (which would be a big overhead). It is not uncommon to have separate social media accounts for a personal project vs personal page, even if the project is not that of a group.

[duck57]

@azbulutlu Thanks for coming up with a much more "professional" use case than I thought of. One install per server makes much more sense than one install per domain. Perhaps you could use some NGINX magic to proxy traffic from one domain to the other as an interim solution, but someone with more NGINX experience than I should comment if this is actually feasible as a solution or not.

[DanielHeath]

@duck57 trying to proxy is fraught - you have to correctly rewrite every instance of the domain name in the body of all resources, but you also have to consider whether you're breaking CSP (since you might have changed the body of a resource which has a hash provided).

It can be made to work but it's not something I'd want to rely on, especially if I'll be updating the backend over time (which might introduce new ways for the nginx config to be wrong).

[duck57]

@DanielHeath Something I have seen people do is use cnames, but I'm not sure if that solves this issue or simply allows a user to access multiple accounts on a single instance through an app that assumes one account per instance per user.

[DanielHeath]

If I cname toots.daniel.heath.cc to mastodon.social, and load up http://toots.daniel.heath.cc, I just get a 'welcome to nginx' page.

[raphaeljolivet]

There are plenty of geeks out there running their own servers and not mutualizing things. I guess this impacts badly the performances of the fediverse as a whole : shared inboxes can't be used and popular toots are replicated to many single-user instances.

To me the solution of @semarie is the way to go.
So let's go.

The first step would be to add a field customDomain in the account table, and spot the places where the username @Foo is assumed to be @[email protected]

For instance the user profile page should accept a full handle like :

• http://instance-domain/users/[email protected]
• http://instance-domain/@[email protected]

Once the WebFinger resolution is done, I guess ActivityPub only refer to full URLs for user IDs, so everything should be fine. right ?

Next step should be to support this in user migration. Typically : migration from the default domain to the custom one, possibly on the same instance. How would migration be handled when moving from one provider to another, with the same handle ?

I would be interested to have the point of view of an active developer on this code base, so we could spot limitations and potential issues with this approach.

[DanielHeath]

@semarie I really appreciate the concept of splitting webfinger out from hosting - that's a really nice design that avoids quite a bit of complexity.

[raphaeljolivet]

So ... I have managed use use my custom domain and make it point to a shared instance, without hacking Mastodon
... and, it kinda works ...

I have developed a small python app webfinger-proxy to alias your own domain handles to a shared instance.

I have setup it on my own domain rjo.name.
The handle [email protected] points to [email protected]
See the response of the Webfinger https://rjo.name/.well-known/[email protected]

I have managed to follow this account and send it private messages.
I have another account on the same shared instance, and [email protected] appears as a completely separate remote account though.

I am not sure whether this is only on the UI or if Mastodon now have a duplicate account for this guy ...
I would need some help from the developers to figure this out : Is the handle a primary key ? Or is it the URL ?
Can you see any have adverse effects ?
Do you think the physical migration of accounts from one instance to another would still work ?

I will continue my tests with other implementations :
Pleroma, mobile clients, etc ....

[almet]

Thanks to @raphaeljolivet last message, I took a similar approach, but with an nginx configuration and static files, since I didn't wanted to have a service running just for this.

If you want to do the same, here's a link to my work.

[almet]

This seems to have limitations currently, since there are now ways to define in the settings what our "main" identity is.

I have two accounts : the "front" one which I want others to see and the "back" one. If I interact with users (answer, follow, etc), I should be seen with my front identity, and the back one shouldn't appear anywhere, I don't think that's what's happening here.

[ThibG]

Please do not do that, and wait for solutions to be implemented in Mastodon itself, since it will require cooperation with the hosting instance.

[raphaeljolivet]

Please do not do that, and wait for solutions to be implemented in Mastodon itself, since it will require cooperation with the hosting instance.

Ok fibne, I have added a warning note to my repo :
https://github.com/raphaeljolivet/webfinger-proxy/blob/master/README.md