Mastodon: Replace Paperclip with Shrine

[Gargron]

Some of our processing pipelines deviate from the norm, leading to strange custom-coded transcoders where the flow of information is not entirely clear. Paperclip locks us solidly into ImageMagick, which doesn't let us decide what exactly to do with files. ImageMagick's convert is often the source of CPU surges. If we could replace Paperclip with Shrine, we could use ImageMagick alternatives like https://www.imageflow.io/ or https://github.com/discordapp/lilliput

Unfortunately, switching between them is an involved process. We would need to keep a 1:1 directory/naming scheme to ensure old files remain accessible, we also have all of these cloud integrations people rely on, like S3, OpenStack Swift, and whatever else. We would need to ensure the configurations remain the same. Most annoyingly, Shrine expects a different database schema for storing attachments, which means a complicated data migration process.

Related PRs: #5655

Some starting points:

• http://shrinerb.com/
• http://shrinerb.com/rdoc/files/doc/paperclip_md.html

[mike-burns]

Note that Paperclip provides a mechanism for swapping the thumbnail professor, e.g. https://gist.github.com/booleanbetrayal/7ed33be807da3e6f95ed

This might be an okay intermediate solution.

[rfwatson]

I'd be interested in helping with this change, or at least trying to get an understanding of what would be involved - especially having spent years fighting with both Paperclip and ImageMagick!

What kind of size of data migration would we be talking about on the largest Mastodon instances?

[mike-burns]

Update with some insider knowledge from team Paperclip: we will be publishing detailed tutorials on how to move from Paperclip to ActiveStorage, in the coming month.