Boto3: Pre-signed URL issues (generate_presigned_post and generate_presigned_url)

[pcraciunoiu]

Hi there,

I'm having two issues which I believe are related. I've already spent a few hours trying to figure this out and it looks like a bug. I finally stumbled on a solution that works, but it's not ideal.

Here is the setup:

Doesn't work

1.

s3 = boto3.client(
    "s3",
    aws_access_key_id=self.aws_access_key_id,
    aws_secret_access_key=self.aws_secret_access_key,
)
signature = s3.generate_presigned_post(
    Bucket=bucket,
    Key=key,
    Fields={"acl": "private", "Content-Type": file_type},
    Conditions=[{"acl": "private"}, {"Content-Type": file_type}],
    ExpiresIn=expires_in,
)

Results in signature["url"] = "https://{bucket}.s3.amazonaws.com/"
Should instead be "https://{bucket}.s3.{region}.amazonaws.com/"

Without region, there is a redirect on the frontend which breaks CORS. See also #421.

2.

url = s3.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': bucket,
        'Key': key,
    },
    HttpMethod="GET",
)

This URL always results in SignatureDoesNotMatch. Presumably also because it is missing the region, and it also gets redirected.

For what it's worth, the region I'm testing with is 'us-east-2'

Works

# This needs to be changed for the frontend to upload
signature["url"] = f"https://{self._bucket}.s3.{self._region}.amazonaws.com/"

# This client init config makes the signature work for download.
s3 = boto3.client(
    "s3",
    "us-east-2",  # Not specifying this breaks.
    config=Config(s3={'addressing_style': 'path'})
    aws_access_key_id=self.aws_access_key_id,
    aws_secret_access_key=self.aws_secret_access_key,
)

This is not an acceptable path forward because AWS is retiring the path addressing_style.

Remainder of the code the same.

Any help greatly appreciated! Thank you for your hard work maintaining this library.

[pcraciunoiu]

FWIW I've also looked at #1644 (this comment helped) and #923.

[swetashre]

@pcraciunoiu - Thank you for your post. As per the documentation the response url doesn't include region. This is the expected url signature["url"] = "https://{bucket}.s3.amazonaws.com/"

And for the second example when i run your code i am not getting any error with the latest version of boto3.
It is possible that switching from virtual to path addressing style could fix pre-signed URLs signed using SigV4. For SigV4 the host is signed as part of the signature, which can cause problems for newly created buckets when virtual addressing style is being used. S3 will redirect to a different host for buckets that DNS haven't propagated for, which leads signature mismatch errors as the host is no longer correct.

Hope it helps ad let me know if you have any further questions.

[pcraciunoiu]

@swetashre but the URL always gets redirected, what's the point of generating it without the region?

So you were able to run it with virtual and the file loads in your browser for the presigned URL? I am using the code above with defaults and I'm not sure why it's not working. Can you post a sample of what you get... I tried this a bunch of different times and was not able to get a URL that actually downloads the file for virtual.

[pcraciunoiu]

Also thank you for your prompt response. I really appreciate it!

[swetashre]

@pcraciunoiu - Thank you for your reply. Here is the code snippet with response from generate_presigned_url

import boto3

s3 = boto3.client('s3')
url = s3.generate_presigned_url(
    'get_object',
     Params={
        'Bucket': 'testbucket',
        'Key': 'outfile.txt',
    },
    HttpMethod="GET",
)

Response url:
https://testbucket.s3.amazonaws.com/outfile.txt?AWSAccessKeyId=SRQPO4NMLKJIHNGFEDCB&Expires=1559248814&Signature=JPaP9naftETY516%2BQvcLb4SipVM%3D

[pcraciunoiu]

Yeah, looks like only v4 is supported for this bucket, in us-east-2. I get this:

<Error>
<Code>InvalidRequest</Code>
<Message>
The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
</Message>
<RequestId>9691BDEC22469DFC</RequestId>
<HostId>
JNbnKZ/bUKph5kHTSMMx6FRlkJPkrxj09hOalqRl0MA/BeaO+EJgoECXNeuIJRew98/TvglY2Lk=
</HostId>
</Error>

[pcraciunoiu]

If I do this instead, I get a different error:

import boto3
from botocore.client import Config

s3 = boto3.client(
    's3',
    config=Config(signature_version='s3v4')
)
url = s3.generate_presigned_url(
    'get_object',
     Params={
        'Bucket': 'testbucket',
        'Key': 'outfile.txt',
    },
    HttpMethod="GET",
)

<Error>
<script/>
<Code>AuthorizationQueryParametersError</Code>
<Message>
Error parsing the X-Amz-Credential parameter; the region 'us-east-1' is wrong; expecting 'us-east-2'
</Message>
<Region>us-east-2</Region>
<RequestId>BFD0841A02020B64</RequestId>
<HostId>
tQs0P8RtoCctKzyaIxGe6BpiiWN/yLBz1RheMKb2lqRe6UwPbXI1G5GqZxCdzB4zVrtvq6VurY0=
</HostId>
</Error>

And if I specify the region, then it works. So I may have stumbled upon a better option than doing path access. I'll try that out in the code path and see if it works.

For the URL in the signature, it seems to me like there should be a way to have it include the region to avoid the redirect...

[pcraciunoiu]

OK, that worked. So it seems like for certain regions the region AND the config v4 need to both be specified. Otherwise the signed URLs don't work. Might need to update the docs for generate_presigned_url, or if that's already there I missed it...

As far as the URL signature, I am doing that string replace, but I'm open to better options. I'm curious why you wouldn't want to include it in the signature URL though.

[pcraciunoiu]

Just wanted to follow up that this doesn't work again. I'm not sure what changed, but now I get:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Again, switching to the path style fixes the issue. So the diff is just:

-                config=Config(signature_version="s3v4"),
+                config=Config(s3={'addressing_style': 'path'}),

[swetashre]

@pcraciunoiu - Thanks for the feedback. Are you still getting the error ? There are certain regions which does not support signature version 2 and for that you have to use v4. Here is the link to documentation:
https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html#signature-2-regions-services

It is possible that switching from virtual to path addressing style could fix pre-signed URLs signed using SigV4. For SigV4 the host is signed as part of the signature, which can cause problems for newly created buckets when virtual addressing style is being used. S3 will redirect to a different host for buckets that DNS haven't propagated for, which leads signature mismatch errors as the host is no longer correct.

Are you still getting error when you specify both signature version and region ? Can you please provide the debug log ?

[pcraciunoiu]

@swetashre yes I got the error when I specify signature version and region. I wasn't able to reproduce it, so I'm not sure what's going on :) Maybe I'm crazy.

[swetashre]

@pcraciunoiu - Thanks for the reply. I am not able to reproduce your issue. It would have been very useful if we had gotten the debug log. Please make sure you are using correct region where the bucket exists and that region supports the specified signature version.

At any way it is working fine for you now.

[arcivanov]

Again, switching to the path style fixes the issue. So the diff is just:

-                config=Config(signature_version="s3v4"),
+                config=Config(s3={'addressing_style': 'path'}),

@pcraciunoiu Thing is, s3v4 is going to be the only mechanism starting June 20th as s3v2 is being decommissioned.

[pcraciunoiu]

Thanks, not so relevant to the issue posted. I think the problem is caused by addressing_style sometimes, and specifying region and or signature v4 seems to fix the URL for us-east-2. This is not documented that I can tell, and it's not obvious what to do to fix when you encounter it.

[swetashre]

@pcraciunoiu - Thanks for all the feedback. I assume this issue has been solved for you. So i am closing this. Please reopen if you have any questions.

[ConnorWhalen]

fwiw i am also seeing this issue. Until a newly created bucket's global DNS gets set up, presigned URLs generated with generate_presigned_url return a redirect and fail CORS. Specifying the region and s3v4 don't fix this but path addressing does, though path addressing will be retired for new buckets next september. Returning the region-specific virtual address would fix this problem, and there doesn't seem to be a way to do that currently through this api. It's also only a problem until the bucket's global DNS gets set up, however long that may take (for me today in ca-central-1 it's ~2 hours)

[revmischa]

Does this work?

s3 = boto_session.client("s3", endpoint_url=f"https://s3.{region}.amazonaws.com")

[getup8]

@revmischa, I had a similar issue (I wanted virtual, regional URLs in my presigned URL), and used your setting but also needed to explicitly specify virtual paths in the config.

import boto3
from botocore.client import Config

region = 'us-east-1'

s3 = boto_session.client(
    's3',
    endpoint_url=f'https://s3.{region}.amazonaws.com',
    config=Config(s3={'addressing_style': 'virtual'}))