Cookiecutter-django: Selecting whitenoise still requires AWS setup

[yunti]

I had selected whitenoise from the django cookiecutter configuration, but when it comes to deploying on heroku I get an improperly configured error if eg DJANGO_AWS_ACCESS_KEY_ID isn't set (same with the other AWS envs). The settings file also outputs the comment at the top: `Use Amazon's S3 for storing static files and uploaded media' even though the whitenoise middleware looks to be properly setup.

[griff-rees-RIT]

Similar confusion, but I'm starting to think this is meant to cover any files uploaded that are separate from css/svg etc which would be loaded in a standard request live via whitenoise. Just a guess, not sure what else it's for and annoying getting around errors. I've enabled heroku and docker to keep options open, so I wonder if that adds both by default, but the heroku documentation specifically refers to using AWS so I guess it's required by default. Would also expect that to relate to data stored via a link field in a Postgres table (for example).

[yunti]

Yes but there also seems to be cache settings for AWS eg AWS_EXPIRY

[griff-rees-RIT]

Yeah I think I miss-guessed on this :P It seems like a halfway house of whitenoise and AWS set up by default, and I'm now trying to get away from AWS and running into some problems. I'm wondering if the generated documentation for deployment isn't configuration dependent (realise I could look this up fairly easily and should). That would be a nice feature to avoid steps that don't fit the configuration choices.

[griff-rees-RIT]

This was my original guess: http://stackoverflow.com/questions/5016589/what-is-the-difference-between-static-files-and-media-files-in-django

[pydanny]

If you have this working, please consider submitting a pull request.

[arthur-wsw]

I already removed django-storage and S3 config from cookiecutter-django generated project and make it working with or without whitenoise (not on heroku but i think it should work to).

Maybe the best thing to do will be to add a variable on project generation something like use_aws_as_media_storage

If this suggestion looks pertinent to you, i think i could submit a pull request by the end of the week.

Keep me in touch

[griff-rees-RIT]

I've still not solved this and suggested fixes would be much appreciated.
I've fought a halfway house of django-storages-redux, AWS and whitenoise under
Heroku, and left it for the moment for focus on other stuff. Thanks all for
help and suggestions.

[pydanny]

@arthur-wsw, a pull request to fix this problem world be nice. But I'm not certain about adding a new option to the project. The reason is that it's inconceivable to me to use whitenoise and a CDN.

For what it's worth, without a pull request to fix this issue I plan to remove whitenoise support from the project.

[yunti]

@pydanny Could you expand a bit further on why? Whitenoise and CDN (not inc AWS) seem to work pretty well.

[jayfk]

Whitenoise is working pretty well in the current configuration on multiple platforms. If we are going to remove it, we have to find another library that supports all of them, too.

From my POV, this is a documentation problem:

• whitenoise is used for static files
• django-storages with AWS is used for media files

That makes total sense, because static and media files both have different requirements. Static files can be generated from scratch for each deployment, media files OTOH are ideally stored in a central place with proper backups (S3).

For certain deployments, it might make sense to add an option to let whitenoise also handle the media files.

[arthur-wsw]

After a little more search, my suggestion will totally breaks Heroku support because there is no persistent storage on Heroku as explained here http://stackoverflow.com/questions/36631648/media-files-on-heroku.
It seems that S3 (or similar offer) is the only way to go in Heroku for serving django media files .

@pydanny why remove whitenoise? It's the recommanded way to go for serving static files with django on Heroku https://devcenter.heroku.com/articles/django-app-configuration#whitenoise and current configuration works pretty well in all case.

Edit: As jayfk said i think the problem is more a documentation issue, maybe we should precise clearly that S3 is required in all case.

[arthur-wsw]

@jayfk Using whitenoise for media files is not supported by whitenoise as explained in documentation http://whitenoise.evans.io/en/stable/django.html#serving-media-files

[jayfk]

@jayfk Using whitenoise for media files is not supported by whitenoise as explained in documentation http://whitenoise.evans.io/en/stable/django.html#serving-media-files

That's correct, but there is an addon for whitenoise available that allows that. If I remember correctly it's called rednoise.

[arthur-wsw]

Correct https://github.com/ryanmcgrath/django-rednoise/ but project seems to be dead (no activity since January)

[pydanny]

@yunti, as to why Whitenoise + CDN makes not sense to me, here we go: If I'm going to go to the trouble of having a CDN serve out my media files, why not also have it serve the static assets? Why add the complexity of two ways of serving media? Hence why Whitenoise + CDN doesn't make any sense to me.

What I think would make the most sense:

1. Keep Whitenoise, but separate it 100% from the AWS S3 option. You have one or the other.
2. Mention in the documentation, and perhaps as a post-hook, for those who choose Heroku+Whitenoise, that they cannot upload media without modifying the production settings file.

[griff-rees-RIT]

There's also CloudFront

For what it's worth I may just try rednoise for the moment.

[mikemccllstr]

I'm only a beginner here, but the second paragraph on the WhiteNoise docs seems to tout that it works with a CDN to replace S3:

It’s designed to work nicely with a CDN for high-traffic sites so you don’t have to sacrifice performance to benefit from simplicity.

It seems like there are are two orthogonal capabilities that will probably be served by separate elements of a solution, once a site grows enough to justify it:

1. Static files, created by the site developer, pushed with the rest of the app code for the site, invalidated on a push. This seems best addressed by WhiteNoise.
2. Media files, often user-generated content that "originate" in production, and rarely changing. In the AWS orbit, S3 seems like the natural place for these.

Either of these capabilities may or may not warrant a CDN in front of them, depending on the needs of the site.

Maybe this complexity is more than cookiecutter-django is intending to address in a starter project template, with all the different possible solutions and how hosting options play into the choices. In that case, the approach @pydanny suggests makes plenty of sense.

I especially like the idea of his second point, where a page in the documentation could suggest "future growth considerations", beyond the template. Otherwise users might mistake the intentional limitations of cookiecutter-django as "the one true way to do things".

[yunti]

@pydanny. Thanks for the input. I think the assumption here is that all sites need to serve media files, where for a lot, they don't so a simplified Whitenoise (plus CDN) does have value.

The original question was assuming no media files were required and was never about whitenoise being used to serve media files (that really doesn't seem to be it's intended use case) . sorry if this wasn't clear.

The default setup when choosing whitenoise almost works with this, it just needs a WHITENOISE_MAX_AGE to be set to work with with a CDN. (All the AWS settings are then redundant if not using S3 for static files).

So perhaps cookiecutter should have a 'use_whitenoise' y/n and a 'use_aws_for_media' y/n question where if whitenoise is selected all awss option to do with static file serving are dropped eg AWS_EXPIRY and if 'use_aws_for_media' is n then the rest of the settings are dropped.

I don't mind if you wish to get the cookiecutter config options down to have this in the docs.

Happy to make the changes needed as so far with django-cookiecutter I've done,

• Whitenoise, static only
• Whitenoise + CDN (static)
• Whitenoise + CDN (static) + AWS (media)

(Probably should have ditched whitenoise in the last case, but it evolved to that overtime. )

[griff-rees-RIT]

@yunti hey any chance you could document your solution? It would be really helpful. Thanks!

[browniebroke]

Took a look at this old issue and there was a few things mentioned, however both cookiecutter-django and Whitenoise have lived since it was open, so let me unpack my finding.

Original issue

I looked at the history of the repo at the time when this issue was open (27th of October 2016) and there was indeed a problem with media setup when selecting Whitenoise:

https://github.com/pydanny/cookiecutter-django/blob/ff582dc4d064676bb345d69c69e31c9f0346ad36/%7B%7Bcookiecutter.project_slug%7D%7D/config/settings/production.py#L123-L133

The DEFAULT_FILE_STORAGE setting wasn't set properly as it is now:

https://github.com/pydanny/cookiecutter-django/blob/f60b6d464cf8fc1751894a0d4756858aad9a680e/%7B%7Bcookiecutter.project_slug%7D%7D/config/settings/production.py#L95-L98

So this part seems resolved

WHITENOISE_MAX_AGE setting

The doc says:

Time (in seconds) for which browsers and proxies should cache non-versioned files.

Versioned files [...] are detected automatically and set to be cached forever.

The default is chosen to be short enough not to cause problems with stale versions but long enough that, if you’re running WhiteNoise behind a CDN, the CDN will still take the majority of the strain during times of heavy load.

Conclusion

I don't think there is any remaining problem left to fix here, so I'm going to close this. If you feel like this is not a true statement, you're welcome to open a fresh issue.