Boto3: Using headers in presigned urls for S3 "get_object" methods.

[random1st]

S3 use "ContentType" header for signature calculation, but client's method has not parameter "ContentType".

[JordonPhillips]

Are you seeing request failures? You can set the response content type with the ResponseContentType parameter (docs).

[random1st]

If I get S3 object by generating url and add "Content-Type" to headers I get "SignatureDoesNotMatch" error.
If i use ResponseContentType parameter, url has parameter response-content-type in self. I tried to append headers manually (create request_dict, add headers, call get_presigned_url method of requestsigner), but url is invalid.
Now I use old boto generate_url and it work correct. Please do not close this issue I think it is important bug. If you need quick response, contact with me immediately.

[random1st]

u"https://test-bucket.s3.amazonaws.com/key/path?Signature=nxK%2BwZ69tQ7NHb0genrCgwW66Yg%3D&Expires=1389139200&AWSAccessKeyId=" - old boto generated; u'https://test-bucket.s3.amazonaws.com//key/path?response-content-type=application%2Fjson&AWSAccessKeyId=&Expires=1389139200&Signature=GcSTuhqKQAaoEfzaFnHMDCyNBVw%3D' -with "ResponseContentType";u'https://key.s3.amazonaws.com/path?AWSAccessKeyId=&content-type=application%2Fjson&Expires=1389139200&Signature=MfFpVaUNCK32CAbUNhd4vjHRmtI%3D'- when I try to create request_dict and add headers manually. I removed AWSAccessKeyId, it isn't error

[random1st]

My code with RequestSigner:

bucket_name = "test-bucket"
key="/key/path"
request_dict= {'query_string': {}, 'headers': {"Content-Type":content_type}, 'url_path': u'/{bucket}/{key}'.format(
                    bucket=bucket_name, key=key
                ), 'body': '', 'method': u'GET'}
            prepare_request_dict(request_dict, endpoint_url="https://s3.amazonaws.com")
            presigned_url=client.instance._request_signer.generate_presigned_url(request_dict, expires_in=URL_EXPIRATION_TIME)`

Result: u'https://test-bucket.s3.amazonaws.com//key/path?AWSAccessKeyId=&content-type=application%2Fjson&Expires=1389139200&Signature=XIYG0hDsdZMUrHDN6rwmR4Z2P%2BI%3D'

[JordonPhillips]

So you're generating the pre-signed URL, and then when you use that URL you're adding Content-Type?

[random1st]

I tried to generate url with content-type header.
`conn=boto.connect_s3()
presigned_url = conn.generate_url(
expires_in=URL_EXPIRATION_TIME,
method='GET',
bucket=bucket_name,
key=key,
headers={'Content-Type': content_type},
)``
It's  old code

[random1st]

If I redirect 302 to S3 presigned url with boto3, request has content-type header. And I have error.

`

SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.
{removed}
GET

application/vnd.api+json
1462287921
{removed}/2cfdb961-0d74-4047-9340-ce3af40cc3bf/2016-04-26-14-57.json
pYQyUnK1KfsV3fcdFU7M57YaVBs=
47 45 54 0a 0a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 76 6e 64 2e 61 70 69 2b 6a 73 6f 6e 0a 31 34 36 32 32 38 37 39 32 31 0a 2f 63 6f 6c 6c 65 63 74 72 69 75 6d 64 65 76 2d 61 74 74 61 63 68 6d 65 6e 74 73 2d 74 65 73 74 2f 61 70 69 36 5f 72 65 73 70 6f 6e 73 65 73 2f 32 63 66 64 62 39 36 31 2d 30 64 37 34 2d 34 30 34 37 2d 39 33 34 30 2d 63 65 33 61 66 34 30 63 63 33 62 66 2f 32 30 31 36 2d 30 34 2d 32 36 2d 31 34 2d 35 37 2e 6a 73 6f 6e
95A13282ED0B5C9E
/0qANbC1yFu6wrlg8G2xETyhwGPsimHUMa6BiMxbPQs6cYYqyT3VuY0tYB/nPm2NcSnLQhRBzac=

`

[random1st]

S3 service calculate signature with headers, boto3 ignore/do not support headers when it generate presigned url

[JordonPhillips]

Is there any reason you're sending ContentType? Requests that do not have a body (GET) do not need to have ContentType set. boto3 will sign all headers except for accept and user-agent.

[random1st]

If I redirect request to S3 (like as 302,301) from API gateway, I cant remove this header from request. And client (browser or other) send headers to redirect url. I don't know any reason why S3 calculate signature with headers when it check signature of request but I think it is correct for S3. And AWS SDK should provide this feature for request's signing.

[JordonPhillips]

Well the issue is that we do sign the header, and we shouldn't be doing that, regardless of the reason why it's there. I'll have a PR up to address this shortly.

[JordonPhillips]

Thanks for raising the issue!

[random1st]

If I understand correctly, you have added Content-Type to your ignore list headers. But you and so do not use it, and S3 his checks. In this case S3 Service should not use this headers, not you

[random1st]

Sure, I can be do not send 'Content-Type' header for request, but API Gateway required this header for request's integration. What can I do if I send POST or another request to another server with Content-Type header and redirect it to S3?

[hdn8]

Any update on this? I set the ContentDisposition to 'attachment; filename: ..' so it can be used as a download but then I get the SignatureMismatch error.

[JordonPhillips]

@hdn8 You will need to provide content disposition as part of your signed request. You could also be having issue with unicode characters, which we don't support in headers for sigv4.

[random1st]

What about ContentType header?

[escanzano]

Is there a bug when the ContentType parameter is included? In the code below, the first block make a signed URL that S3 accepts. But when I specify a ContentType, S3 rejects the PUT with 403. Does anyone know if there is an issue here?

return S3Utils.s3_client.generate_presigned_url(method, Params={ 'Bucket': config.s3_bucket_name, 'Key': key }, ExpiresIn=config.s3_expire_seconds, HttpMethod='PUT')

return S3Utils.s3_client.generate_presigned_url(method, Params={ 'Bucket': config.s3_bucket_name, 'Key': key, 'ContentType': content_type }, ExpiresIn=config.s3_expire_seconds, HttpMethod='PUT')

[random1st]

@escanzano You should use boto (old AWS SDK) generate_url to fix it. I think this is AWS S3 bug and it won't be fixed in boto3 code.

[random1st]

conn = boto.connect_s3() presigned_url = conn.generate_url( expires_in=url_expiration_time, method='GET', bucket=self.bucket_name, key=key, headers={'Content-Type': content_type}, )

[Jon-Omnibond]

Can we please have the ability to add headers to pre-signed urls in boto3 just as is demonstrated above in the old boto methods?

I have a situation where I'm using a request to API gateway to run a Lambda function which generates a data file. This file is then uploaded to s3 since it is too large to return directly via the gateway. I generate a signed url and return a 302 redirect with this url as the location.

It works fine in boto3 as long as the customer doesn't call the api endpoint with a Content-Type header but if they do that header causes the presigned url to be invalid as it is passed along with the redirect. This is very much a real world situation where a Content-Type header would be passed to a pre-signed s3 'get' request url (even though it isn't needed it is there from the original API request).

[random1st]

There is no reaction from developers for two years. I use boto in case when I need to generate url with headers

conn = boto.connect_s3() presigned_url = conn.generate_url( expires_in=3600, method='GET', bucket=bucket_name, key=key, headers={'Content-Type': content_type}, )

[radix]

I was able to generate a presigned url with this code and it worked.

        url = S3.generate_presigned_url(
            ClientMethod='get_object',
            ExpiresIn=60,
            Params={
                'Bucket': bucket_name,
                'Key': key_name,
                'ResponseContentDisposition': 'attachment;filename={}'.format(filename),
                'ResponseContentType': 'application/octet-stream',
            }
        )

I didn't get any SignatureMismatch error.

[harshgangwar]

Thanks radix,
Your solution helped me to resolve the issue.
'ResponseContentType': 'application/octet-stream'

[swetashre]

@random1st - Sorry for following up on this so late. There is no parameter for custom headers in boto3's generate_presigned_url. The headers get added based on the parameters that you would normally pass to the client's method. So something like this:

import boto3
import botocore


s3_con = boto3.client('s3')
url=s3_con.generate_presigned_url('get_object',
                                  Params={'Bucket': 'bucket_name',
                                          'Key':'img.jpg',
                                          'ResponseContentType': 'image/jpg'
                                  },
                                  ExpiresIn=600)
print(url)

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.get_object
I hope it helps. Please let us know if it is still an issue for you.