Boto3: s3 copy_from fails if "key" has spaces in it.

[jeffmacdonald]

I'm writing a lambda handler that takes event notifications from a remote S3 bucket and copies a newly uploaded object into "local" bucket.

Here is the important code:

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    try:
        s3.Object(bucket + '-clone', key).copy_from(CopySource=bucket + '/' + key)

When I upload a file to execute the code I get the following error in Cloudwatch:

An error occurred (AccessDenied) when calling the CopyObject operation: 
Access Denied: ClientError Traceback (most recent call last): 
File "/var/task/lambda_function.py", line 15, in lambda_handler raise e ClientError: 
An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

It seems that a space gets translated to a + somewhere along the way. I added the following
key = key.replace("+", " ") to my code, and now it copies properly without error.

[JordonPhillips]

It seems like this has something to do with url encoding since all those params go into the url or into headers. Spaces should get translated as %20 though, so that's a bit strange. Could you post a debug log so I can get a better idea of whats going on under the hood (as I am unable to reproduce)? You can enable debug logging with boto3.set_stream_logger('botocore').

[jamesls]

Closing due to inactivity.

[seanpaulkelley]

I am also having the same problem. A key with a space in it fails to get retrieved.

for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
file_name_pattern = re.match( r'(.*?([^/]+)/?)$', key)
file_name = file_name_pattern.group(2)
download_path = '/tmp/{}{}'.format(uuid.uuid4(), file_name)
s3_client.download_file(bucket, key, download_path)

My work around is to set key like
key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'].encode("utf8"))

I added the stream loggger and here is my output

2016-12-21 17:48:48,423 botocore.vendored.requests.packages.urllib3.connectionpool [INFO] Starting new HTTPS connection (1): my-photos.s3.amazonaws.com
[INFO] 2016-12-21T17:48:48.423Z b97124d2-c7a5-11e6-b626-712e694e52b4 Starting new HTTPS connection (1): my-photos.s3.amazonaws.com
2016-12-21 17:48:48,630 botocore.vendored.requests.packages.urllib3.connectionpool [DEBUG] "HEAD /9d781651-f5b9-445c-b238-721e0e581297Hannah%2BTrot.JPG HTTP/1.1" 404 0
[DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 "HEAD /9d781651-f5b9-445c-b238-721e0e581297Hannah%2BTrot.JPG HTTP/1.1" 404 0
2016-12-21 17:48:48,630 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'rp4ws0BtuN07AbdxRcN4JOM3BNg/kHFOixWhgPQL0PmUnNN7em3AP1I2n+PUHvl9j3ncpEeyPg4=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '24BB44067C63C2EE', 'date': 'Wed, 21 Dec 2016 17:48:47 GMT', 'content-type': 'application/xml'}
[DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 Response headers: {'x-amz-id-2': 'rp4ws0BtuN07AbdxRcN4JOM3BNg/kHFOixWhgPQL0PmUnNN7em3AP1I2n+PUHvl9j3ncpEeyPg4=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '24BB44067C63C2EE', 'date': 'Wed, 21 Dec 2016 17:48:47 GMT', 'content-type': 'application/xml'}
[DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 Response body:

[DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 Event needs-retry.s3.HeadObject: calling handler
[DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 No retry needed.
[DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 Event needs-retry.s3.HeadObject: calling handler >
2016-12-21 17:48:48,630 botocore.parsers [DEBUG] Response body:

2016-12-21 17:48:48,631 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler
2016-12-21 17:48:48,631 botocore.retryhandler [DEBUG] No retry needed.
2016-12-21 17:48:48,631 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler >
An error occurred (404) when calling the HeadObject operation: Not Found: ClientError
Traceback (most recent call last):
File "/var/task/createThumbnail.py", line 47, in handler
s3_client.download_file(bucket, key, download_path)
File "/var/runtime/boto3/s3/inject.py", line 125, in download_file
extra_args=ExtraArgs, callback=Callback)
File "/var/runtime/boto3/s3/transfer.py", line 269, in download_file
future.result()
File "/var/runtime/s3transfer/futures.py", line 71, in result
return self._coordinator.result()
File "/var/runtime/s3transfer/futures.py", line 231, in result
raise self._exception
ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

END RequestId: b97124d2-c7a5-11e6-b626-712e694e52b4

[moijes12]

Facing this same issue.

[charanjit-singh]

Me too while Accessing the File

[charanjit-singh]

From AWS S3 Web Interface:

URL: /Real Estate/ is encoded as: /Real%2520Estate/

---

From Jupyter Notebook:

When I simply use /Real Estate/:

Raises No Such Key Error

---

From Jupyter Notebook using Web Interface's encoding:

When I use /Real%20Estate/ as input:

Same No Such Key Error!!!

How to process names with spaces in BOTO ???

[mikeatlas]

I also encountered this issue; key = key.replace("+", " ") before making the boto call fixed it for me.

[chilcom]

You probably meant
key = key.replace("%20", " ")
That worked for me

[trulearn]

Neither '+' or '%20' works for white space(" ") of Key in s3.head_object (boto3 1.16.10)
Need help to resolve