Synapse: Need to be able to safely cleanup local content

[rkfg]

Not the caches, the content itself. People sometimes upload big files like videos or highres photos and they stay in the storage forever. Currently there's no way to find these files and delete them after some time safely. The problem is that avatars (for example) are also stored in the same storage and just deleting everything older than N days would effectively purge every user and room avatar if they're set a while ago. I also noticed that purging history with the API doesn't delete the media attached to the purged events. In the end the storage would grow indefinitely which is quite a problem.

So I would like those files to be removed if no events use them (including the forwarded ones), that's how it should be. I know that redacting doesn't actually remove the content from the database just yet but it's unreachable and the media should be too.

But also I'd like to be able to remove media using some criteria like date range and file size and be sure that it doesn't break anything. Sure, easy find -size ... -delete would do but this really should be managed by synapse itself.

[rednerd]

I think the way to solve this right now is to delete files older than X days and larger than Y bytes (e.g. find /var/lib/matrix-synapse/media -size +50M -mtime +365). I have the same issue because my private homeserver on a tiny 10GB VPS is 3/4 full of videos of kids. I've used this method to clean up some files and it just results in a blank message in Riot from what I can see. (I can't get the purge API to actually purge, but that's a separate issue)

If there's a better way, I'm happy to be corrected.

[rkfg]

You can do that but the database records linking to that content would not be removed. Same problem with the local thumbnails, you need to remove the database rows or else the thumbnails won't be regenerated on access later. I think there should be a single cleanup API that takes care of all the local and remote content, including thumbnails and URL previews.

[mytbk]

I wrote a simple tool to clean up local content tonight.
https://git.wehack.space/matrix-synapse-scripts/tree/mxclean

[Linuxine]

I agree, it would be great to have a way to clear the local content without destroying all avatars. I have created a shell script to clean this repository, but as said by @rkfg, this also delete the avatars, even if I save and restore their files (because the local thumbnails are not regenerated after the clean).

[temandroid]

I wrote a simple tool to clean up local content tonight.
https://git.wehack.space/matrix-synapse-scripts/tree/mxclean

This is solution for pgsql, do u have one more for sqlite? :))

[cryzed]

We made synapse-purge to purge remote media cache, local media and events in all rooms (both encrypted and unencrypted) until "x seconds ago" for the Synapse server, feel free to use it. It is highly configurable and should work on all instances using a Postgres database. This also preserves only the most recent user and room avatars and will remove old ones.

[ShadowJonathan]

Possibly related to #890 and #2315

[anoadragon453]

Hello there. The next major release of Synapse should include an admin API to delete local media by timestamp of last access (which helps exclude avatars) and file size. This was introduced by #8519, with documentation on how to use it located here.

I'm going to close this issue now. Feel free to open another issue if you have any further requirements :slightly_smiling_face:

[Linuxine]

Hi, thanks @anoadragon453 for the response, and the great feature ! I was really waiting for this one :D

I tried using the API to delete local media (using -X POST https:///_synapse/admin/v1/media//delete?before_ts=xxx) but I get a response

{
    "errcode": "M_UNRECOGNIZED",
    "error": "Unrecognized request"
}

Is it implemented in 1.22, or will it be in a next version ?
Thanks a lot again !

[clokep]

This will be in 1.23.0.

[Linuxine]

Ok, thanks @clokep ! I was being too impatient, sorry :D