Wagtail: Large files cannot be uploaded when Wagtail is hosted on Heroku

[alexgleason]

This might be outside the scope of Wagtail, but I'm assuming it isn't, since I have seen Heroku listed as a concern in several places and by several core members.

In ProjectTIER/projecttier.org#5 I discovered that Heroku has a 30 second timeout for all requests. In other words, if any request does not complete within 30 seconds, it will stop. This means any file upload that takes over 30 seconds to complete will be aborted and the file cannot be uploaded.

Because hosting large files is necessary for the project, this leaves me with three options:

1. Migrate the Wagtail site away from Heroku
2. Force Wagtail to upload the files differently
3. Teach the client a workaround

I'm torn on what to do here. If you all think it's worth doing, I might explore making option 2 work.

Exploring option 2:

One way is to make Wagtail's admin UI upload direct to S3 using JavaScript. Basically, clicking "upload" causes the browser to connect directly to the S3 instance and upload it. It happens entirely on the front-end. Once it's complete, a JS callback could submit the data to Wagtail. This feature could be toggled on with a setting.

A second way might be to break up the requests in some way. So, when I upload the file, it transfers maybe 5mb per request then Wagtail concatenates it on the server side. I don't actually know if this is possible or how to go about it. Resumable uploads?

Option 3:

Another consideration is to have the client upload these files directly using an S3 GUI like Cyberduck. However, I do not think merely uploading the file will cause it to appear in the Documents pane of Wagtail. It needs a database entry associated with it. So I'd have to write a script, additionally, that scans for changes in S3 and updates Wagtail's database. Possible, but not ideal.

[gasman]

Hi @alexgleason,
I don't think this really belongs in Wagtail - we have enough problems of our own to solve without trying to find workarounds for limitations in third-party services, and supporting large-enough-to-break-Heroku documents seems like a fairly niche requirement.

This would be better handled as a third-party module - would be happy to consider any customisation hooks to the document upload forms to make this possible.

[stevejalim]

@gasman Here's how my former team at YJ solved the direct-upload problem: https://github.com/yunojuno/django-s3-upload -- though it's not going to be a drop-in solution for wagtailadmin, I think

[tomdyson]

@alexgleason thanks for raising this and outlining the options. FWIW I think Heroku's limitation

• is sensible
• will impact a high proportion of Wagtail users

so I'm keen that we find a solution, whether it's a third party module (e.g. @stevejalim's link above) or something more integrated into Wagtail itself.