Boto3: S3 'key' encoding inconsistent within boto3

[hwine]

Context: Lambda function triggered on "put" events, trying to download the reported object.

The value for the 'key' reported in the Lambda event can not always be passed directly back in as the value for 'key' in an S3 API function. Rather, you must perform an encoding incantation that is only revealed in the generated blueprint code for Python S3 events:

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'].encode('utf8'))
    ...

(This can occur when the uploaded keys contain spaces, which S3 encodes with '+' when generating the event. But the boto3 s3 calls expect a string with the actual spaces.)

One could reasonably argue that this is an API design issue, but I've grown soft and expect my API wrappers to hide those implementation details from me.

[joguSD]

This is not possible for us to handle automatically as it's impossible to accurately detect url encoding. I'm not really sure if that's what you're asking for though. We don't control the s3 event api or the lambda handlers so there's really not anything we can do here. I would try asking the s3 or lambda team on AWS support.

[hwine]

Ahh, given that boto3 is installed by default for python lambda functions, I thought boto3 was providing the shim that mapped the "raw lambda" parameters into the 'event' and 'context' objects.

You're right -- if that isn't boto3, you can't do anything. :( At least this might serve as a something that turns up in a search engine for the next person.

[joguSD]

Closing this out here as there's no action item for us. Let me know if you have any other questions.