Matrix-doc: Consider IPFS as an alternative to MXC for Matrix's file repository (SPEC-223)

[matrixbot]

Currently we use our basic mxc:// distributed URI scheme for sharing file attachments between matrix servers. This is a very naive system where you PUT files to your local HS, and then users in the room on remote HSes request the file from their local HS, which downloads and caches a copy on their behalf in order to decentralise the contents.

There are some problems here:

• No streaming file transfer (yet)
• Caching semantics of the remote copies are up to that HS; it's relatively easy for all the HSes to decide to expire the content and lose the original forever
• It has no mechanism for discovering which servers have cached copies of a file, so once the originating server discards its copy there is no way for it or another server to recover a file.
• There is no built-in check to confirm that a file downloaded from a server matches the file that was uploaded. For example we don't include the hash of the file in the mxc uri.
• Your HS needs storage for all the files it's consciously cached
• It has no mechanism for grouping files together into a hierarchical namespace or otherwise labelling files so that they could be accessed by desktop tools.
• ...?

It's an interesting thought experiment to consider whether supporting IPFS URLs as well as MXC URLs in m.file (and similar) events would help with this. Clients which aren't IPFS-aware would need to go via an IPFS bridge; ones which are running an IPFS node would be able to go use IPFS itself.

Thoughts?

(Imported from https://matrix.org/jira/browse/SPEC-223)

(Reported by @ara4n)

[matrixbot]

Jira watchers: @erikjohnston @NegativeMjark @ara4n

[matrixbot]

(IPFS is http://ipfs.io and currently basically a P2P decentralised DAG-based global filesystem, a bit like S3 or similar - but evolving into a potential replacement for the web itself in terms of a decentralised place for arbitrary hypertext and anything could live: https://github.com/ipfs/ipfs. It's evolving to support storing arbitrary JSON content, and could sprout realtime JSON transfer behaviour in future).

-- @ara4n

[matrixbot]

• No streaming file transfer (yet)

Since IPFS splits large files into smaller chunks it looks like it should be possible to stream files, that is have one node download a file while it is still being uploaded.

-- @NegativeMjark

[matrixbot]

A few questions:

• How would we generated thumbnails for content hosted on IPFS?
• How easy is it to block illegal content from a server?

-- @NegativeMjark

[matrixbot]

Will IPFS allow us to do ACLs?

-- @erikjohnston

[matrixbot]

@​markjh There are plans for a blacklisting service to block illegal content from participating nodes: https://github.com/ipfs/gateway-dmca-denylist
@​erikj IPFS doesn't yet have a permission model, but it's a planned feature.

-- David Roberts

[Mikaela]

• ...?

• Everything on IPFS is public even if it may start as unlisted (IPFS Privacy by Pinata at Medium) so this can easily be unexpected behaviour for unencrypted direct chats. I think the idea of publishing encrypted data on IPFS would also be strange in case the encryption was broken in the future.
• Data on IPFS may not be removable if others than the homeservers (after they run gargace collection) pin it, which may be a problem with GDPR
  
  
  
  • even if MXC currently doesn't remove anything (https://github.com/matrix-org/synapse/issues/1263), but at least the removed files aren't publicly available anywhere?
  
  

Otherwise I love the idea and have suggested similar to a lot smaller project (https://github.com/FruitieX/teleirc/issues/262).

Quick edit: I think IPFS would be a great idea for stickers (https://github.com/vector-im/riot-web/issues/2950) assuming they are like at Telegram so anyone can add them and they are public and there isn't a legitimate reason to remove them entirely (which I think would result to unhappy users).

[swedneck]

Recently i've learnt of some new projects building ontop of IPFS to fix the access control issues:
https://textile.io
https://blog.textile.io/ipfs-experiments-creating-ipfs-links-that-you-can-delete/