Mastodon: Support Post Migration

[SilverWolf32]

177 – Support Account Migration – was closed after implementing follower migration, but this is only one small part of a true migration. To really be able to change instances, you need to be able to take your posts with you. There was some good discussion on this over there; I'm opening a new issue to make it clear that this is a separate concern from that issue, which seemed to evolve into only being about followers.

Personally, I couldn't care less about migrating my followers/following list. I can refollow people and have them refollow me. It'll all work out. My posts, however, are currently _impossible_ to restore.

[MirceaKitsune]

Once more I fully support this and have been eagerly awaiting the much requested feature. I would love to see the ability to import the data archives (favorites boosts and toots) exported from the https://instance.social/settings/export section, which is currently a one-way system only. Unfortunately there doesn't seem to be a plan for this yet, but I'm staying hopeful that will change eventually. I believe Eugen mentioned performance concerns on the previous discussion.

[trwnh]

Reiterating points from previous discussion:

• statuses are currently attached to an account, and need to be decoupled first
  
  
  
  • acct is currently authoritative over database id, so all internal logic needs to be reworked to use database id instead (#10745)
  
  
• status move should ideally be as simple as updating some records (after above points are addressed), but this can still be a heavy operation for the database
• status import cannot be backfilled for performance reasons

[SilverWolf32]

Thanks @trwnh!

status import cannot be backfilled for performance reasons

What's backfilling?

[trwnh]

backfilling is #34 -- fetching old statuses from a profile that were never delivered to the instance. you can't expect to deliver 100,000 activities every time you migrate, because that's wildly inefficient.

[MirceaKitsune]

Regarding performance and backfilling: This is why I believe that post migration is possible, but those who choose to use it will need to accept a long waiting period. There will probably have to be three types of limitations to make it viable:

1. An account queue. For example, the instance may only allow up to 5 accounts to be migrated simultaneously. The archive import menu should show a message saying something like "you have to wait until 10 more accounts finish migrating before we start the import process".
2. Every post and action will be migrated with a given delay. This could be one post per 10 seconds. The import menu should then show the message "10 out of 1000 posts migrated" to let you know about the status of the procedure.
3. Only a certain number of the latest activities will be imported. If an archive contains 1.000.000 actions, the interface informs the user that only the latest 100.000 will be considered.

Obviously, if you're importing something like 10.000 boosts + faves + toots, the process may take days if not weeks. It's a price the user will have to pay. But in this format it sounds like it should be performance viable at least.

[SilverWolf32]

That makes sense! Personally I'm perfectly fine with them taking a long while to move over, so long as they move over _eventually_.

I'm not quite sure what the limit should be, though – one post per 10 seconds seems _awfully_ slow to me, unless you have like 100 accounts being migrated at once. Maybe set it relatively fast by default (like 1/s or maybe even a little higher), since most instances are small and won't need to deal with a lot of migrations, then larger instances can set it slower?

[SilverWolf32]

Although I would _really_, really prefer if everything gets migrated, even if it takes weeks or even months. I don't want to lose my old posts – that's basically an archive of what I'm like, so new people can get to know me a little better before they decide whether to follow me. It's also _history_, and while some may not want to keep that sort of thing, others (like me) would.

(Even if you technically have everything in the backup, that's incredibly cumbersome to look at, given I'm not aware of any software that can actually import it.)

[trwnh]

stuff can get imported, but it is absolutely a bad idea to redeliver it. i am firmly of the opinion that a Move with post import should maybe do a rewrite of existing statuses in the database, but nothing more. it is unwise to even consider backfilling until the actual decoupling and importing gets implemented.

more to the point in general, though: it is fundamentally bad idea to assume that every instance should have a copy of every single post at all times. we don't need to duplicate everything 6000x or more. what is fundamentally happening with a "migration" is that the authority is being shifted; the location is being shifted, that's all. it's a Bad Idea to treat them as new statuses. hopefully i've explained why

[SilverWolf32]

Hm...well, would they be new statuses? I figured they would simply be reassigned, but then maybe redelivered _as the originals_, not duplicated and "reposted".

Or maybe I'm completely misunderstanding how federation works. I know very little about the backend.

every instance should have a copy of every single post at all times. we don't need to duplicate everything 6000x or more

Is this actually being discussed? I'd say for a migration the situation is a bit different; only the destination server has to receive a copy of every single post, not _everybody_.

[trwnh]

only the destination server has to receive a copy of every single post

yeah, and it should be limited to this.

looking at prior art from zot, you can do an online migration (fetch old messages from your outbox and inbox to the new server) or an offline migration (using an exported account archive that contains at minimum your keys, your posts, and your address book)

[SilverWolf32]

That sounds lovely! I've been a little worried in the back of my mind about how to handle a server that just suddenly _dies_.

[lmachucab]

Is there at least a first-party tool that users (not servers) can use to work their downloaded backup and generate something usable? Something like a HTML file (plus media?) that can be uploaded into a remote server. Would at least temporarily solve the problem of making the lost content accessible again.

[MirceaKitsune]

@lmachucab I made a NodeJS script that I used to port my faves and boosts to other accounts. Unfortunately it no longer worked last time I tried it, and it was a crude solution that put quite a bit of stress on the target server and even triggered an instance's flood protections once. If you think you might have some use for it still, it's still available on my Github:

https://github.com/MirceaKitsune/mastodon_migrate

[masstransithonchkrow]

Hi there! Mastodon.cloud literally has a month to live. We're being shut down because the domain is being forced to bow to insane regulations, and the admin isn't having any of it.
https://mastodon.cloud/@TheAdmin/104227496654670309

We had been the target of immense DDoS attacks for the past six months.
If you need help or simply wish to deliberate, please let me know.

[msikma]

I'd like to second that Mastodon.cloud disappearing shows this is a crucial feature. Mastodon is a great piece of software, and designing it to be decentralized was a great idea. Making it possible for anyone to set up an instance, yet having instances be able to communicate with one another, makes the network resilient and eliminates single points of failure.

But without the ability to jump from one instance to another at will, this decentralization has a large drawback. If an instance decides to call it quits, you're just going to lose your data. This also means that as a user, you can't be sure if the instance you picked is going to live a long or a short life. Given how many instances there are, it's totally reasonable to expect that some of them are going to disappear.

Aside from users choosing to migrate somewhere else, there should also be a migration plan for instance admins. If an instance shuts down, it shouldn't just be that active users who care about their account get to keep their data, because that means even if user migration is possible there's still going to be a big loss of online culture and data. I think this should probably be a separate issue and come with a host of problems by itself, but without it I think we'll see Mastodon content regularly disappearing forever as instances come and go.

[MirceaKitsune]

Yeah even I didn't expect this sad news. mastodon.cloud banned me randomly and without explanation after a few months of using it... at the time I was upset, now I'm glad I escaped this event. It's sad to hear even Japan is becoming a dangerous place for internet freedom and has a government failing to understand technology and the flow of information.

I know mastodon.cloud was one of the largest instances out there. If it can go down, the fediverse as a whole is at serious threat of significant data loss. That's why I would further implore that we please support FULL account migration now... even if as an option disabled by default which the instance admin can enable if they so wish, in case performance is such a big concern.

An equally better idea would be a node-based approach to storage, similarly to how IPFS / DAT in concept; Instead of an instance being stored on and served by one physical server, it can be stored in a decentralized cloud where anyone may run a node to serve a copy of the data. Sadly this would require such a rewrite that we'd be talking about a whole new project over the existing Mastodon.

[masstransithonchkrow]

A reminder that we're supposed to "Own Your Own Data With Mastodon!" If we don't find a way to do that with toots, we are bound to be sued for false advertising. I smell a frivolous lawsuit that's bound to cripple the Fediverse and we need to get ahead of it.

While I may not ideologically agree with Gab, which accounts for 25% of the Fediverse's mass, they have every right to be here as we do, and we need to prepare for pointed media smear campaigns that could potentially harm our reputation.

We need to emphasize our content controls, and remind users that they can ban who they want from their own console, and that moderators do not have to do it for them. We have that advantage over Twitter and we need to advertise th out of it.

[Valenoern]

I smell a frivolous lawsuit

Mastodon's own code is under the AGPL, right?
as I remember, it and the GPL have some fairly thorough sections saying you (gab) can't turn around and sue a project you built on (mastodon) because it didn't fit your use case, and that if they can't cancel liability/warranty, courts are asked to "arrange something as close to that as possible".

That's the situation you're talking about, right?

I'll agree the AGPL would have no effect on media outlets saying whatever charged things about mastodon they want to, though.

[masstransithonchkrow]

That is an interesting point. I think they forked Mastodon, though (as of 2.8.5) and created Gab out of it. Ironically, it seems they were able to preserve their "toots" during the migration.

My comment revolved around the idea that you can "Own Your Own Data With Mastodon". If this is the case, we must ensure that people have the capability to export their toots otherwise that slogan could be misconstrued as false advertising. We must have a provision that allows users to migrate to another server if the one they're on is being shut down through no fault of its own (regulatory crackdowns in the case of Mastodon.cloud).

What happened in Minneapolis is a distraction. The attacks against the Fediverse and all other entities like it will intensify. We must fortify our infrastructure and address any outstanding issues so that we're ready for any exodus.

[masstransithonchkrow]

Even a static representation of said toots is fine with me, similar to Twitter's archive, which you can navigate offline. They don't need to be integrated into the post chronology. Perhaps we could create a designation for such toots, and also prohibit their editing or removal to ensure a smooth migration.

[ccoenen]

there's an issue for that https://github.com/tootsuite/mastodon/issues/9461 (export including some bare html-version of your toots)

With regard to european GDPR, there's the rule that you must have access to a machine readable version of you data. This is satisfied already. While this satisfies the letter of this particular law, I do wish it was easier for everyone to use that data (hence the ticket above). I also think re-importing would be nice.

Yet, I do not believe there's any grounds to your "false advertising" point. Sorry.

(edit: I linked to the wrong issue before, this has been corrected.)

[masstransithonchkrow]

Thank you for clarifying that. I'll follow that issue too. ^_^

[CoWinkKeyDinkInc]

I strongly support this. I have my own independent instance that I don't want to maintain anymore so I moved over to another instance. DigitalOcean doesn't let you download complete images as a backup so I can't wait to upgrade to a later version that supports it and transfer the old content over.

[lmachucab]

This is a feature that it's impressive that Mastodon does not have in 2020, honestly. Without the ability to export and import post history not only can accounts be lost but also identities - as even if an account is migrated to another server, at the moment this is a partial "in name only" process. Any content that the old account had posted is still bound to the lifetime of the old service (and its FQDN).

After having tested some scripts for extracting the information from the exported data dumps, I think there's enough of those scripts and projects wandering about that it's perfectly feasible to grab one (1), integrate it into Mastodon, and offer it as an utility that eg.: generates full sessions / scripts for clients like toot to execute, resulting in a toot post or toot upload of parts of or of the whole archive in HTML or similar readable format.

[mattbk]

@lmachucab do the existing scripts deal with not treating the imported posts as new posts that would be picked up for federation? From what I'm reading above, that's what should be avoided to reduce load on servers.

I would be happy if my posts were only viewable from my account page/feed rather than immediately (or ever) federated (i.e., they would only be federated if someone was looking at my account). All I want to be able to do is change my username or move to another instance if needed.

[masstransithonchkrow]

I thought I'd bring this up:

One of the SPC admins revived bofa.lol, a previously discontinued instance, only for it to regurgitate its old content over TWKN, some of which was excessively vile. bofa.lol aside, I'd like to focus on the mechanism that caused those toots to refederate and see if we can harness that mechanism to archive them for other instances.

As of this writing, bofa.lol was taken down less than a day after it was briefly revived.

*Edited for context and some spelling mistakes at 21:50-0500