Mastodon: Import toots from CSV

[ddevault]

Toot import should be fairly straightforward, but the concern becomes that it could allow abuse. I think the following constraints on imported toots should be sufficient:

• Imported toots just show up in your profile, not in the local or federated timelines
• Only toots in the past can be imported

Follow-up to #978, cc @Gargron for buy-in

[Natanji]

Doesn't this open the door for forgery?

As I see it, if I run an instance, I want to assure users that something posted there was actually posted at that specific time. If my users trust me as the server admin, they can be sure that the toots weren't forged since there is no way to alter them after the fact.

But now you introduce the import mechanism, and the data provided by the user can of course easily be altered. Suddenly, that guarantee goes out the window. That seems problematic.

Therefore, I believe we need a way to forever flag imported toots as imported and potentially untrustworthy, at the very least.

[tobypinder]

Imported toots could have a metadata field that ties them back to the source toot Uri maybe? Doesn't help if the old instance deleted but it provides some provenance and the UI could choose whether or not to display the timestamp based on whether it matches the source toot or not...

[ddevault]

As I see it, if I run an instance, I want to assure users that something posted there was actually posted at that specific time. If my users trust me as the server admin, they can be sure that the toots weren't forged since there is no way to alter them after the fact.

I don't think this is particularly important. I think it's more important to give a user full control over their account data. You couldn't impersonate another user and I don't think it's a huge deal that you could add toots in the past. Perhaps we should limit you to doing this once?

[ddevault]

Could also just have a visible indication in the UI of whether or not a toot was imported.

[enkiv2]

Forged imported toots have the potential for abuse in the context of spam if they aren't subject to the same post frequency restrictions as regular toots. I'm more concerned about someone importing a ten million spam toots fake-dated to the past ten seconds than someone fudging their past in other ways.

Re: once-only import -- I'm opposed on the grounds that it prevents someone from merging the histories of three or more previously-separate accounts, which seems like it might be kind of a common situation with all the node-hopping that's happened recently.

[Natanji]

@SirCmpwn I think it matters a great lot if you can add toots in the past. You could use this as proof that you said something to another user that they didn't see, which without any kind of indication can be used for a huge lot of malicious purposes, like repudiation or gaslighting. Other users might not know that importing toots is even possible, or that unlike e.g. Twitter, they cannot trust the info. Social networks run on trust, in any case - that runs coutner to any possibility of importing toots so freely.

I propose a different avenue: There is a dicussion in #256 about editing toots after the fact. The same concerns that apply here obviously apply for editing, and importing toots coupled with the existing ability to delete toots would mean that the edit functionality is possible. It is therefore only logical that before any import is possible, the discussion in #256 needs to be finished to allow edits.

Once it is possible to edit toots, we could use the same system for imports: a toot would show "last edited at xy", where xy would become the data at which the toot was imported. This way it is clear that the information may have changed/not have been there originally, while the status can still be shown to have been originally tooted at the original time.

[ddevault]

I did just say that two comments up. Do people just not read?

[dallasread]

Just a thought:

• When you export, a checksum of the export could be stored on the original instance.
• When importing, you must specify the origin instance of your file. The new instance could then fire off an API request to the origin instance to verify its validity.

[ddevault]

Easily defeated by just running your own instance to forge toots from.

[enkiv2]

Being protected from actively malicious nodes you are federated with is a
different, harder problem. Ideally, we should come up with something that
would still be reasonable if every node other than the local one was
possibly actively patched to be malicious.

Federation doesn't protect someone from building a new node from scratch
that produces an entirely fake history. So, nonrepudiation will be hard
without making breaking changes to the protocol. I would suggest that
nonrepudiation is something that will be very difficult in general for
opensocial & that it should be addressed separately from importing history,
simply because any system for nonrepudiation will have to be layered on top
of inter-node federation if it's going to be useful, and will probably need
to be applied even to completely unrelated codebases that speak the same
protocol. (I.e., it might be something akin to PGP on top of email -- a
front-end change that injects hashes into the timeline -- or a separate
service, like some kind of oracle that keeps a hash chain of toots.)

That said, there are big vulns that are easily fixed with bulk import,
unrelated to nonrepudiation.

On Thu, Apr 6, 2017 at 10:14 AM Drew DeVault [e-mail]
wrote:

Easily defeated by just running your own instance to forge toots from.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/981#issuecomment-292187580,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAd6GVDV1IqJv4jKb5qOyIO2vUO5-7iFks5rtPNSgaJpZM4M0hL2
.

[AdmiralRazorbeard]

An idea:

Each instance simply generates a unique private key and defines its own list of acceptable instances for importations

When an instance exports a list of toots, there are two additional fields in the CSV:
-originOfExport
-generatedChecksum which is equal to _checksum(checksum(toots) + originOfExport + private key)_

When you want to import your toots to a new instance, this new instance sends

• checksum(toots) (or the list of toots instead of checksum(toots) ) calculated from this new instance
• generatedChecksum extracted from the CSV
• originOfExport extracted from the CSV

to the instance corresponding to the originOfExport address which checks the validity with its private key.

If the origin is wrong -> rejected
If the checksum is wrong -> rejected
If the original server is down -> the admin of this new instance decides if he accepts new importations without verification

This way the trick "running your own instance to forge toots from" just can't work.
This way the server doesn't have to store a list of checksum.
This way you can't cheat with the originOfExport.

It could be quite easy to implement imho and we could even imagine websites providing lists of instances of trust to help admin keeping their own list up to date.

[enkiv2]

Doesn't this mean that any instance is expected to keep a whitelist of
instances they'll allow imports from? That encourages a small group of big
instances that all support cross-imports -- bad for federation.

Federation already makes forgery easy. (For instance, you can run a fake
node that emits forged toots & follow it from a real node -- which may in
fact be easier than modifying a real instance.) More important than
avoiding forgery (which could be solved by front-end changes like toot
signing) is making sure mass import can't circumvent existing posting
rules; after all, any given node has restrictions on post frequency, and
import makes forging post frequency easy -- and any other automatic
restriction on posts (say, content blacklists) should also be applied.

On Thu, Apr 6, 2017 at 2:45 PM ThomasFournet [e-mail]
wrote:

An idea:

Each instance simply generates a private key and defines its own list
of instances of trust

When a server export a list of toots, there are two additional fields in
the CSV:
-originOfExport
-generatedChecksum which is equal to checksum(checksum(toots) + private
key + originOfExport)

When you want to import your toots to a new instance, this new instance
send

• checksum(toots)
• generatedChecksum
• originOfExport
  to the previous instance which check the validity with its private key.

If the origin is wrong -> rejected
If the checksum is wrong -> rejected

If the original server is down, the admin of this new instance decides if
he accepts the new importation.

This way the trick "running your own instance to forge toots from" can't
work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/981#issuecomment-292272501,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAd6GfwsBiREt3xZK9KgapL34qqI2JLFks5rtTLDgaJpZM4M0hL2
.

[AdmiralRazorbeard]

It could be better to have a blacklist of nodes instead of a whitelist so everyone is allowed by default. I don't think it's bad for federation. It's just great to avoid jokers.

[Natanji]

Blacklists solve absolutely nothing at all because anyone who is
blacklisted just needs one more DNS or IP, which are plentiful and cheap.
Seriously, that is just not an option.

I don't think it's bad if you are locked into a smaller amount of nodes but
still have a choice where you wanna be, for this very feature. Importing
toots is not a killer feature, please keep in mind that most users will not
need it and never even consider using it! But all users will be all
negatively affected by unfiltered information and forged toots. Giving up
the few assurances a healthy federated community is able to Mastodon users
will simply destroy Mastodon completely.

ThomasFournet [e-mail] schrieb am Sa., 8. Apr. 2017,
17:53:

It could be maybe better to have a blacklist of nodes instead of whitelist
so everyone is allowed by default. I don't think it's bad for federation.
It's just great to avoid jokers.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/981#issuecomment-292726619,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA2J9gaSXW5IZdSy4SgSl0oJmMGOvxIgks5rt61vgaJpZM4M0hL2
.

[sandhawke]

Can we prototype this and explore the social implications with a command line script the admin runs, to put the posts directly into the database? Will that work without any code changes? It wont have a visible indicator the posts were imported, but maybe that's not so important. I think there's some problem, though, where the code deeply assumes posts are added in time order.

[renyuneyun]

I read this thread and find no conclusion has been made.
For me, a person out of mastodon's development cycle, I feel an indicator is quite enough by saying:

1. This content is imported
2. When is is content imported
3. When it claimed to be created

That should provide enough information for a people to judge whether to trust the (existence of) toot based on the trust of the instance owner, or to trust the (existence of) toot based on the people who own that account.
This trust behaviour is the same as if there is no such importing tool.

Not sure if it adds extra usefulness by saying "this imported content is claimed to be imported" (i.e. export & import two times).

As other people have said above, it makes no point to "verify the trustworthy of the content" by connecting with another instance, both by a) it could have been dead already; b) it may not be trustworthy.
Similar thing also applies when it comes to cryptography (and hence everything), as long as there is no way to provide an unalterable signature including time. From my knowledge, I don't think any existing cryptography can do this (I would imagine maybe quantum has the potential, but that's not a near future).
Unless people want to introduce hierarchical trust (like SSL), there is no way to "verify".

[sandhawke]

Actually secure timestamping is super easy, barely an inconvenience. (Okay, slight exaggeration for a joke, but yeah, it's doable. Dunno if it's worth it for this.)

https://gist.github.com/Manouchehri/fd754e402d98430243455713efada710

https://en.wikipedia.org/wiki/Trusted_timestamping

[renyuneyun]

@sandhawke So that still requires a "trusted third party", isn't it?

Personally I would say this is useful as an extension, but should not be used as the core functionality. (But I'm not in mastodon development team.)

[sandhawke]

Yes, but everything requires trusting somebody/something. Something is "secure" if the cost/difficulty of breaking it is more than that of alternative paths to some end.

I'd assume if the server admin trusts the user, they can accept timestamps from them. If they don't know and trust the user, then they'd need one of these external systems to provide "secure" timestamps before allowing back-dating of posts. How secure a service they require depends on how secure they want timestamps to be on their instance - I imagine any of these professional services would be fine for current Mastodon instances.

[stephane-klein]

👍 for this issue.

I would like migrate from Twitter to Matodon and I would like to import my Tweet archive to my Mastodon instance.

[yumetodo]

When really need proof that toot was tooted at the time, you should use external service like WebArchive.
To avoid spam attacks:

• make an interval time for importing
• reject toots that were posted future.
• add a label to notice which was imported.

All another problem pointed out here cannot be resolved I think because Mastodon is Federation.

I think that Add a signature to exported status ( #12649 ) doesn't resolve any problems. Because all users can be some server owners and they can manipulate databases directly.

[davr]

Here's a script someone wrote to do the import if you control the destination database:
https://gitlab.com/Simounet/mastodon-toots-migration-queries-generator

There's nothing preventing this currently, so any arguments about how people can't trust the network if we let people import are invalid, because it's already possible, it's just a huge hassle. Please consider making this an officially supported feature doable via the UI instead of requiring commandline and database access.

[umonaca]

I've forked the script above to make it compatible with Mastodon v3.2.0. Also added a command to revert changes to the database.
https://gitlab.com/umonaca/mastodon-toots-migration-queries-generator