Boto3: presigned URLs require redirect, breaking CORS

[wichert]

I am trying to implement a direct upload feature for a website. On the backend I generate a presigned URL so I can do the upload from the browser:

bucket = self.request.registry.settings['s3.media.bucket']
client = boto3.client('s3')
url = client.generate_presigned_url('put_object',
    Params={
        'Bucket': bucket,
        'Key': '%s/%s' % (data['category'], data['key']),
        'ACL': 'public-read',
        'ContentType': data['content_type'],
    },
    ExpiresIn=300,
    HttpMethod='PUT')

The resulting URL follows the standard naming pattern of https://.s3.amazonaws.com/. When I try to do the file upload from a browser the browser will first issue a CORS pre-flight OPTIONS request. This results in a 307 temporary redirect from AWS to https://<bucket>.*<region>*.amazonwas.com/<path>. Since redirects for CORS requests are not allowed the browser immediately aborts the request.

Is there a way to generate a URL that will not result in a redirect using boto3?

[jpluscplusm]

Can you perhaps specify a Region key in the Params struct?.
(I'm just guessing here, based on some awscli use which had the same problem and effectively the fix I've mentioned.)

[wichert]

Hm, possibly. Looking at botocore source that means I will have to use boto3.client('s3', region). I'll give that a try.

[wichert]

Interestingly the region name appears to be stripped if it is a known region:

>>> import boto3
>>> params = {'Bucket': 'jzoo-media', 'Key': 'my/file', 'ACL': 'public-read', 'ContentType': 'image/png'}
>>> boto3.client('s3', 'ap-southeast-1').generate_presigned_url('put_object', params, ExpiresIn=300, HttpMethod='PUT')[:70]
'https://jzoo-media.s3.amazonaws.com/my/file?Expires=1451749838&Signatu'

And trying the same thing with a bogus region:

>>> boto3.client('s3', 'invalid').generate_presigned_url('put_object', params, ExpiresIn=300, HttpMethod='PUT')[:70]
'https://s3.invalid.amazonaws.com/jzoo-media/my/file?X-Amz-Credential=A'

[Chronial]

Yes, this is rewritten by this function: https://github.com/boto/botocore/blob/729057f3c4753e49c07c1d3fd233bd200be6db65/botocore/utils.py#L645

I have the same problem and would like to generate region-specific urls to avoid the CORS issue.

[JordonPhillips]

@wichert Sorry for the delay on my part. S3 supports CORS, but only if you specifically configure the bucket to do so. You can do this via the web console under the permission section of the bucket, or through the sdk.

[Chronial]

No, that is not the problem. We are talking about cases in which the CORS was set up correctly. CORS is only guaranteed to work if you use the region-url. For non-region urls it only works after some time has passed since bucket creation. Here is some explanation on that:
https://github.com/component/s3/issues/7#issuecomment-45581857

Waiting will always fix the issue, but it would be nice if boto3 could generate urls that also work for new buckets.

[JordonPhillips]

@Chronial In that case, you should either sign the request with sigv4 or remove that function from event handler. I'll include this use case in #425 since it's a bit of a gotcha. For now, here's how to use sigv4:

import boto3
from botocore.client import Config

s3 = boto3.client('s3', 'us-west-2', config=Config(signature_version='s3v4'))
s3.generate_presigned_url('list_buckets')
u'https://s3-us-west-2.amazonaws.com/?~~etc~~~'

[Chronial]

Thx, that documentation fixes the issue for me. I saw in the code, that another signature version might work, but had no idea how to switch version.

[DavyCode]

I am using AWS sdk for uploads, after spending some time searching online i stumbled upon this thread. thanks to @lsimoneau 45581857 its turns out the exact same thing was happening. I simply pointed my request Url to the region on my bucket by attaching the region property and it worked.

const s3 = new AWS.S3({
accessKeyId: config.awsAccessKeyID,
secretAccessKey: config.awsSecretAccessKey,
region: 'eu-west-2' // add region here
});

[pcraciunoiu]

It looks like this was fixed supposedly, but I'm seeing redirects today on boto3==1.9.156. This happens even if I try the sigv4 versioning or specify the region as per @JordonPhillips ' example

The redirect adds the region to the domain/URL. If I add this manually from the backend, the frontend works just fine. Any help would be appreciated.

signature["url"] = f"https://{self._bucket}.s3.{self._region}.amazonaws.com/"