Boto3: SimpleDB put_attributes raises SignatureDoesNotMatch error when UTF-8 value is given

[robert-nash]

I am successfully using boto3 to save data to SimpleDB however, if I try to store a string which contains UTF-8 characters, the bellow error is thrown.

Traceback (most recent call last):
  File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/python/run/venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/python/current/app/application.py", line 269, in articleAdmin
    'Replace': True
  File "/opt/python/run/venv/lib/python2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/python/run/venv/lib/python2.7/site-packages/botocore/client.py", line 395, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutAttributes operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

[rayluo]

Would you mind to provide a minimal boto3 code snippet together with your UTF-8 test string to demonstrate the problem? By the way, not all UTF-8 characters are supported by SimpleDB. Please refer to its limitation.

[andrewreece]

I ran into this issue too. It looks like it can be fixed if the request['Content-Type'] field can be adjusted to include the utf-8 charset. Is this an available setting for the sdb boto3 client?

[samoconnor]

I've run into this problem too trying to build a simple AWS Lambda function to do an SDB PUT.

@rayluo here is an example:

sdb.put_attributes(DomainName = 'foo',
                   ItemName = 'foo',
                   Attributes = [{'Name': 'foo', 'Value': 'bär', 'Replace': True}])

ClientError: An error occurred (SignatureDoesNotMatch) when calling the PutAttributes operation:
The request signature we calculated does not match the signature you provided.

The V2 signature signs a normalised query string. Maybe the percent-encoding of the UTF-8 character is wrong in this query string??

My Julia code for SDB works ok using this v2 signing code: https://github.com/samoconnor/AWSCore.jl/blob/master/src/sign.jl#L30

[samoconnor]

Below is some debug logging...

The problem seems to be the Content-Type: header.
As @andrewreece mentioned above, it should be application/x-www-form-urlencoded; charset=utf-8 see https://github.com/samoconnor/AWSCore.jl/blob/master/src/sign.jl#L41

send: 'POST / HTTP/1.1\r\n
Host: sdb.ap-southeast-2.amazonaws.com\r\n
Accept-Encoding: identity\r\n
Content-Length: 651\r\n
Content-Type: application/x-www-form-urlencoded\r\n
User-Agent: Boto3/1.3.1 Python/2.7.10 Linux/4.1.19-24.31.amzn1.x86_64 Botocore/1.4.24\r\n
\r\n
SignatureVersion=2&
AWSAccessKeyId=XXX&
Attribute.1.Value=b%C3%A4r&Timestamp=2016-06-14T23%3A12%3A51Z&
Attribute.1.Replace=true&
DomainName=foo&
SecurityToken=XXX&
SignatureMethod=HmacSHA256&
Version=2009-04-15&
Signature=XXX&
Action=PutAttributes&
Attribute.1.Name=foo&
ItemName=foo'

[samoconnor]

This nasty hack fixes the problem:

from botocore import endpoint
def make_request(self, operation_model, request_dict):
        request_dict['headers']['Content-Type'] = 'application/x-www-form-urlencoded; charset=utf-8'
        return self._send_request(request_dict, operation_model)
endpoint.Endpoint.make_request = make_request

[samoconnor]

@jamesls, are you the right person to make a quick fix for this?

It looks like the Content-Type comes from here https://github.com/boto/botocore/blob/7384364df5b7f438678ad0777f07ce37c39e9f59/botocore/vendored/requests/models.py#L460

The SimpleDB documentation makes it clear that charset=utf-8 is required:
http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/MakingRESTRequests.html#POSTRequestStructure

[samoconnor]

bump
@JordonPhillips ?

[randxm]

A quick hack for @samoconnor's hack above, so as to only affect requests headed towards the sdb endpoint:

from botocore import endpoint
def make_request(self, operation_model, request_dict):
    if self._endpoint_prefix == 'sdb':
        request_dict['headers']['Content-Type'] = 'application/x-www-form-urlencoded; charset=utf-8'
    return self._send_request(request_dict, operation_model)
endpoint.Endpoint.make_request = make_request

[simplesteph]

@samoconnor @randxm you saved my day! Would love for boto3 to incorporate the fix on the headers

[memory]

Believe it or not there are a few SDB users out there still; it would be nice to get this fixed.

[JordonPhillips]

We would be willing to accept a pull request to fix this if this is still a problem for people. Otherwise we're not likely to get to it due to prioritization. :(

[memory]

Fair enough; let me see if I can get a passing PR together.