Boto3: Error "An error occurred (404) when calling the HeadObject operation: Not Found" while accessing S3 Object fields

[keane777]

Describe the bug
Unable to get S3.Object fields for "directory" object only if it was created via boto3 API.

When we create directories manually in AWS S3 Web-Console everything works fine.

Important
We have two user accounts with AdministratorAccess permission:

• First for s3-bot and programmatically access.
• Second for user and manual manipulations with S3 Service.

Steps to reproduce

This leads to ERROR 404

1. Create clear bucket (for example, "test-bucket") manually in AWS S3 Web-Console.
2. Upload file with two sub-directories in path (for example, "first/second/file.txt"):

s3_resource = boto3.resource(
        "s3", 
        aws_access_key_id="my_access_key_id", 
        aws_secret_access_key="my_secret_access_key", 
        region_name="us-east-1"
    )

bucket = s3_resource.Bucket(name="test-bucket")
bucket.upload_file(Filename=r"c:\test\file.txt", Key="first/second/file.txt")

1. Try to get object of "first/" directory and print it's content length:

first_dir = bucket.Object(key=r"first/")
print(first_dir.content_length)

1. Here comes error botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found.

This work's fine

1. Create clear bucket (for example, "test-bucket-2") manually in AWS S3 Web-Console.
2. Create manually nested directory structure in your new bucket - "first_dir/second_dir/".
3. Try to get object of "first_dir/" directory and print it's content length:

s3_resource = boto3.resource(
        "s3", 
        aws_access_key_id="my_access_key_id", 
        aws_secret_access_key="my_secret_access_key", 
        region_name="us-east-1"
    )

bucket = s3_resource.Bucket(name="test-bucket")
first_dir = bucket.Object(key=r"first_dir/")
print(first_dir.content_length)

1. Now we have our "content_length" value.

Expected behavior
A clear and concise description of what you expected to happen.

Debug logs

2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2020-05-22 17:49:04,527 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2020-05-22 17:49:04,543 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2020-05-22 17:49:04,543 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2020-05-22 17:49:04,543 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2020-05-22 17:49:04,543 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2020-05-22 17:49:04,574 botocore.loaders [DEBUG] Loading JSON file: C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\boto3\data\s3\2006-03-01\resources-1.json
2020-05-22 17:49:04,574 botocore.loaders [DEBUG] Loading JSON file: C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\botocore\data\endpoints.json
2020-05-22 17:49:04,574 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x0000027DB613FF28>
2020-05-22 17:49:04,626 botocore.loaders [DEBUG] Loading JSON file: C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\botocore\data\s3\2006-03-01\service-2.json
2020-05-22 17:49:04,631 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x0000027DB6116B70>
2020-05-22 17:49:04,631 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x0000027DB636AB70>
2020-05-22 17:49:04,653 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x0000027DB6116950>
2020-05-22 17:49:04,654 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2020-05-22 17:49:04,655 botocore.loaders [DEBUG] Loading JSON file: C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\botocore\data\_retry.json
2020-05-22 17:49:04,656 botocore.client [DEBUG] Registering retry handlers for service: s3
2020-05-22 17:49:04,656 boto3.resources.factory [DEBUG] Loading s3:s3
2020-05-22 17:49:04,657 boto3.resources.factory [DEBUG] Loading s3:Bucket
2020-05-22 17:49:04,657 boto3.resources.model [DEBUG] Renaming Bucket attribute name
2020-05-22 17:49:04,657 botocore.hooks [DEBUG] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x0000027DB63B2510>
2020-05-22 17:49:04,658 s3transfer.utils [DEBUG] Acquiring 0
2020-05-22 17:49:04,658 s3transfer.tasks [DEBUG] UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000027DB6934780>}) about to wait for the following futures []
2020-05-22 17:49:04,658 s3transfer.tasks [DEBUG] UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000027DB6934780>}) done waiting for dependent futures
2020-05-22 17:49:04,659 s3transfer.tasks [DEBUG] Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000027DB6934780>}) with kwargs {'client': <botocore.client.S3 object at 0x0000027DB68AC588>, 'config': <boto3.s3.transfer.TransferConfig object at 0x0000027DB69340B8>, 'osutil': <s3transfer.utils.OSUtils object at 0x0000027DB6934080>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x0000027DB69342B0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x0000027DB6934780>}
2020-05-22 17:49:04,659 s3transfer.futures [DEBUG] Submitting task PutObjectTask(transfer_id=0, {'bucket': 'tarakhti-test', 'key': 'first/second/file.txt', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x0000027DB69342B0> for transfer request: 0.
2020-05-22 17:49:04,659 s3transfer.utils [DEBUG] Acquiring 0
2020-05-22 17:49:04,659 s3transfer.tasks [DEBUG] PutObjectTask(transfer_id=0, {'bucket': 'tarakhti-test', 'key': 'first/second/file.txt', 'extra_args': {}}) about to wait for the following futures []
2020-05-22 17:49:04,659 s3transfer.tasks [DEBUG] PutObjectTask(transfer_id=0, {'bucket': 'tarakhti-test', 'key': 'first/second/file.txt', 'extra_args': {}}) done waiting for dependent futures
2020-05-22 17:49:04,659 s3transfer.tasks [DEBUG] Executing task PutObjectTask(transfer_id=0, {'bucket': 'tarakhti-test', 'key': 'first/second/file.txt', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x0000027DB68AC588>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x0000027DB6934B38>, 'bucket': 'tarakhti-test', 'key': 'first/second/file.txt', 'extra_args': {}}
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x0000027DB61796A8>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x0000027DB6176AE8>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x0000027DB6179F28>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x0000027DB6176A60>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x0000027DB68CFCC0>>
2020-05-22 17:49:04,660 botocore.hooks [DEBUG] Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x0000027DB61766A8>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function conditionally_calculate_md5 at 0x0000027DB61769D8>
2020-05-22 17:49:04,662 s3transfer.utils [DEBUG] Releasing acquire 0/None
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x0000027DB6176D90>
2020-05-22 17:49:04,662 botocore.handlers [DEBUG] Adding expect 100 continue header to request.
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x0000027DB617A0D0>
2020-05-22 17:49:04,662 botocore.endpoint [DEBUG] Making request for OperationModel(name=PutObject) with params: {'url_path': '/tarakhti-test/first/second/file.txt', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.13.14 Python/3.7.3 Windows/10 Botocore/1.16.14 Resource', 'Content-MD5': 'mfivAtEyk6BJ20L8BGF/oA==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x0000027DB6934B38>, 'url': 'https://s3.amazonaws.com/tarakhti-test/first/second/file.txt', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x0000027DB68AC6A0>, 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'tarakhti-test'}}}
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler <function signal_not_transferring at 0x0000027DB66DAAE8>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000027DB68ACA20>>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x0000027DB63F8908>>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x0000027DB6176598>
2020-05-22 17:49:04,662 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x0000027DB68CFD30>>
2020-05-22 17:49:04,662 botocore.utils [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2020-05-22 17:49:04,662 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/tarakhti-test/first/second/file.txt
2020-05-22 17:49:04,662 botocore.utils [DEBUG] URI updated to: https://tarakhti-test.s3.amazonaws.com/first/second/file.txt
2020-05-22 17:49:04,662 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-05-22 17:49:04,662 botocore.auth [DEBUG] CanonicalRequest:
PUT
/first/second/file.txt

content-md5:mfivAtEyk6BJ20L8BGF/oA==
host:tarakhti-test.s3.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20200522T144904Z

content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2020-05-22 17:49:04,662 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200522T144904Z
20200522/us-east-1/s3/aws4_request
589c3fb87a672582117df8c32afc0410a55501ced06b296b5973897af5e1804a
2020-05-22 17:49:04,663 botocore.auth [DEBUG] Signature:
0a50f542fac6771b98a0f3cb4a55e2cb7b7bcc676035389f9cb9a4db3051c140
2020-05-22 17:49:04,663 botocore.hooks [DEBUG] Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x0000027DB66DAB70>
2020-05-22 17:49:04,663 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://tarakhti-test.s3.amazonaws.com/first/second/file.txt, headers={'User-Agent': b'Boto3/1.13.14 Python/3.7.3 Windows/10 Botocore/1.16.14 Resource', 'Content-MD5': b'mfivAtEyk6BJ20L8BGF/oA==', 'Expect': b'100-continue', 'X-Amz-Date': b'20200522T144904Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIATJFUOQC4GVET5CJ5/20200522/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=0a50f542fac6771b98a0f3cb4a55e2cb7b7bcc676035389f9cb9a4db3051c140', 'Content-Length': '21'}>
2020-05-22 17:49:04,663 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): tarakhti-test.s3.amazonaws.com:443
2020-05-22 17:49:05,168 botocore.awsrequest [DEBUG] Waiting for 100 Continue response.
2020-05-22 17:49:05,324 botocore.awsrequest [DEBUG] 100 Continue response seen, now sending request body.
2020-05-22 17:49:05,496 urllib3.connectionpool [DEBUG] https://tarakhti-test.s3.amazonaws.com:443 "PUT /first/second/file.txt HTTP/1.1" 200 0
2020-05-22 17:49:05,496 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'f+mYBYi9cjNIiD583lJwA6/7vm1FT8sZFo5XTbR2zTZcNu6WIRbCp6+wZoenJndTPMjpJ7+wnXs=', 'x-amz-request-id': '8CF3961EB3937A02', 'Date': 'Fri, 22 May 2020 14:49:06 GMT', 'ETag': '"99f8af02d13293a049db42fc04617fa0"', 'Content-Length': '0', 'Server': 'AmazonS3'}
2020-05-22 17:49:05,496 botocore.parsers [DEBUG] Response body:
b''
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x0000027DB68CF978>
2020-05-22 17:49:05,496 botocore.retryhandler [DEBUG] No retry needed.
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
2020-05-22 17:49:05,496 s3transfer.utils [DEBUG] Releasing acquire 0/None
2020-05-22 17:49:05,496 boto3.resources.factory [DEBUG] Loading s3:Object
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event creating-resource-class.s3.Object: calling handler <function lazy_call.<locals>._handler at 0x0000027DB63B2598>
2020-05-22 17:49:05,496 boto3.resources.action [DEBUG] Calling s3:head_object with {'Bucket': 'tarakhti-test', 'Key': 'first/'}
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x0000027DB6176AE8>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x0000027DB6176A60>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-parameter-build.s3.HeadObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x0000027DB68CFCC0>>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x0000027DB61766A8>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x0000027DB6176D90>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event before-call.s3.HeadObject: calling handler <function inject_api_version_header_if_needed at 0x0000027DB617A0D0>
2020-05-22 17:49:05,496 botocore.endpoint [DEBUG] Making request for OperationModel(name=HeadObject) with params: {'url_path': '/tarakhti-test/first/', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.13.14 Python/3.7.3 Windows/10 Botocore/1.16.14 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/tarakhti-test/first/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x0000027DB68AC6A0>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'tarakhti-test'}}}
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x0000027DB66DAAE8>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x0000027DB68ACA20>>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x0000027DB63F8908>>
2020-05-22 17:49:05,496 botocore.hooks [DEBUG] Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x0000027DB6176598>
2020-05-22 17:49:05,512 botocore.hooks [DEBUG] Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x0000027DB68CFD30>>
2020-05-22 17:49:05,512 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/tarakhti-test/first/
2020-05-22 17:49:05,512 botocore.utils [DEBUG] URI updated to: https://tarakhti-test.s3.amazonaws.com/first/
2020-05-22 17:49:05,512 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-05-22 17:49:05,512 botocore.auth [DEBUG] CanonicalRequest:
HEAD
/first/

host:tarakhti-test.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20200522T144905Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2020-05-22 17:49:05,512 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200522T144905Z
20200522/us-east-1/s3/aws4_request
9aecf738d76489a898ce323a23650c1ea2c1d5dc51a95ae364c4c613434770f7
2020-05-22 17:49:05,512 botocore.auth [DEBUG] Signature:
281122febb2c5d1164fb21175080d62ea2cae7d5a91dda2f6acf854e5f32c00b
2020-05-22 17:49:05,512 botocore.hooks [DEBUG] Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x0000027DB66DAB70>
2020-05-22 17:49:05,512 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://tarakhti-test.s3.amazonaws.com/first/, headers={'User-Agent': b'Boto3/1.13.14 Python/3.7.3 Windows/10 Botocore/1.16.14 Resource', 'X-Amz-Date': b'20200522T144905Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIATJFUOQC4GVET5CJ5/20200522/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=281122febb2c5d1164fb21175080d62ea2cae7d5a91dda2f6acf854e5f32c00b'}>
2020-05-22 17:49:05,652 urllib3.connectionpool [DEBUG] https://tarakhti-test.s3.amazonaws.com:443 "HEAD /first/ HTTP/1.1" 404 0
2020-05-22 17:49:05,652 botocore.parsers [DEBUG] Response headers: {'x-amz-request-id': '2F8918091001D969', 'x-amz-id-2': 'VN5/HE64IJmBwjRTTAr7KDGHWzHbnLSH2OMJSRCmS5t2qWE4amQkdpYLRELyxYOGH/iuUMN98ug=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 22 May 2020 14:49:05 GMT', 'Server': 'AmazonS3'}
2020-05-22 17:49:05,652 botocore.parsers [DEBUG] Response body:
b''
2020-05-22 17:49:05,653 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x0000027DB68CF978>
2020-05-22 17:49:05,653 botocore.retryhandler [DEBUG] No retry needed.
2020-05-22 17:49:05,653 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x0000027DB68CF630>>
Traceback (most recent call last):
  File "C:/Projects/Veeam Test/boto3_example/main.py", line 97, in <module>
    main()
  File "C:/Projects/Veeam Test/boto3_example/main.py", line 88, in main
    print(first_dir.content_length)
  File "C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\boto3\resources\factory.py", line 339, in property_loader
    self.load()
  File "C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\boto3\resources\factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\boto3\resources\action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\botocore\client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Projects\Veeam Test\boto3_example\venv\lib\site-packages\botocore\client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

[swetashre]

@keane777 - Thank you for the post and providing all the details. When you create a folder via s3 console, it creates an object with the name appended by suffix "/" and that object is displayed as a folder in the s3 console. So it will be returned on a HeadObject operation.

But if you create the file test1/test2/file.txt that's just the name of the file, it doesn't create test1/ and test2/ for you. Object storage is a flat file structure unlike block storage.

When you create a folder from the console it mentions that behavior as well:

Hope it helps.