Cookiecutter-django: Bug report: Setting up a different AWS S3 region in production.py

[Afrowave]

I work out of Nairobi, so I set up a Digital Ocean "droplet" in London and wanted my static and media files hosted in London as well.

The default settings for AWS s3 are:

STATIC_URL = 'https://s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
and
MEDIA_URL = 'https://s3.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME

These settings would produce a static url for a web site favicon image of
s3.amazonaws.com/staticstore.example/static/images/favicon.ico
where staticstore.example is the AWS_STORAGE_BUCKET_NAME environment variable.

After a number of unsuccessful tries [ like 2 days :-( ], I went to my s3 console and found out that my favicon is being served from s3 in Europe(London) with the "s3.eu-west-2.amazonaws.com/staticstore.example/static/images/favicon.ico" url.

To replicate:
I did 3 things:

1. Created a new vanilla cookiecutter-django project, built it with the local.yml and made sure it has its "static" and "media" files are being served.

2. In production.py, I changed the following settings:

   ```
   ...
   MEDIA_URL = 'https://s3.eu-west-2.amazonaws.com/%s/media/' % AWS_STORAGE_BUCKET_NAME

   # Static Assets
   # ------------------------

   STATIC_URL = 'https://s3.eu-west-2.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
   ...
   ```

3. I built a new s3 bucket whose only change was region set to "EU(London)". I built a new user and created an s3 policy and attached the new user to the s3 bucket.

I destroyed the droplet and built a new one using the changed production.yml. Using the docker-compose -f production.yml up command, I could see all the static files being copied to my new bucket.

Expected behavior:
On running the website, only django html appeared and all the static files were not served. On checking it out, I found that the {% static 'images/favicon.ico ' %} was converted to https://s3.amazonaws.com/staticstore.example/static/images/favicon.ico and yet I expected https://s3.eu-west-2.amazonaws.com/staticstore.example/static/images/favicon.ico

While I can not see where the bug would come from if settings.MEDIA_URL and settings.STATIC_URL were being processed by Django properly, the only other option is the storages.backends.s3boto3 was not reading the production.py configurations but making up its own default url based on some rule.

Can anybody else confirm this?

[orzarchi]

It seems like Boto 3 will sometime rewrite your s3 urls for you to use what is called virtual hosting addressing.

I'm not sure I understood your problem fully, but to avoid this behaviour you can set the django-storages setting AWS_S3_SIGNATURE_VERSION to 's3v4' and see if that helps.

I used it to fix problems with django-compressor: Because these rewrites caused my {%static%} tags to issue urls that didn't contain the s3 address set in COMPRESS_URL, django-compressor complained.

[MightySCollins]

I had ALOT of pain setting S3 up properly. One thing which should work for you is setting AWS_S3_CUSTOM_DOMAIN which you could change to https://s3.eu-west-2.amazonaws.com/BUCKET_NAME.

Have you also tried setting AWS_S3_HOST?. I also think that in the London region it is required to use v4 signatures.

[orzarchi]

I think you are right regarding v4 signature being required, but I'm not sure boto3 switches to use it automatically.

Also django-storages documentation is outdated - they recommend to set S3_USE_SIGV4=True for all new projects, but that doesn't do anything. Only AWS_S3_SIGNATURE_VERSION='s3v4'

[luzfcb]

Try use AWS_S3_REGION_NAME on production settings file.
From django-storages docs:

AWS_S3_REGION_NAME (optional: default is None)
Name of the AWS S3 region to use (eg. eu-west-1)

http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

[MightySCollins]

If it helps here is a small snippet of my config for the London region https://gist.github.com/MightySCollins/ef5a02ef9eea70a681f9a4b2efc61de4. Also this link also helped http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region.

Like I said it was painful... a big problem I remember having with compressor needing access to the files. If there is a nicer way to do this let me know.

[orzarchi]

I had the same journey and you and reached almost the exact same config.
Just a tip:
Your cache headers are not being set with that config, as the setup in cookiecutter is targeted to boto, and not boto3.
Here is the correct setting:

AWS_EXPIRY = 60 * 60 * 24 * 7
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': ('max-age=%d, s-maxage=%d, must-revalidate' % (AWS_EXPIRY, AWS_EXPIRY))
}

[MightySCollins]

@orzarchi Thanks for that, ill test it and get it rolled out. I assume I don't need the entire AWS_HEADERS section then.

[Afrowave]

@luzfcb, @MightySCollins. Thanks. I will check it out and once it works I will submit a PR for adding this bit of information to the "Deployment with Docker" section at least.