Boto3: Handling botocore.exceptions.ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation

[raolivei]

I am trying to handle a botocore exception that occurs when an S3 object is on Glacier. Error is:

botocore.exceptions.ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

After reading https://github.com/boto/boto3/issues/1262 and google search everywhere, it looks like this exception cannot be handled.

I have tried to create an exception clause for ClientError but it looks like the error happened before the object goes to the exception block:

        try:
            storage_class = s3.Object(BUCKET_NAME, KEY).storage_class
        except botocore.exceptions.ClientError:
            if storage_class == "GLACIER":
                print("Remove key", KEY, "from", storage_class)
                pass
        else:
            raise

I have also tried to use ClientError error code as InvalidObjectState (as I could not find a number that represents this exception):

        try:
            storage_class = s3.Object(BUCKET_NAME, KEY).storage_class
            print("Remove key ", KEY, " from", storage_class)
        except botocore.exceptions.ClientError:
            if e.response['Error']['Code'] != "InvalidObjectState": 
                raise

Note that I use != to raise and exception if Error code is different than InvalidObjectState.
Even if I use ==, (or change the next action to pass instead of raise), same error occurs.

The result is below. I can see the message that I print in this case, but if I move print one line down to the except block, it would never show up. What makes me think that the error is happening before we can handle it.

Remove key 00efa84eb2d83a3cb  from GLACIER
Traceback (most recent call last):
  File "./dpkg-parser", line 260, in <module>
    main()
  File "/Users/roliveira/.pyenv/versions/3.7.2/envs/dpkg-log-parser/lib/python3.7/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/Users/roliveira/.pyenv/versions/3.7.2/envs/dpkg-log-parser/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/roliveira/.pyenv/versions/3.7.2/envs/dpkg-log-parser/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

Also in https://github.com/boto/boto3/issues/1262 you can find an Exception hierarchy with a list generated programatically with all exceptions that can be handled – InvalidObjectState is not in the list:

AliasConflictParameterError < ValidationError < BotoCoreError < Exception
ApiVersionNotFoundError < BotoCoreError < Exception
BaseEndpointResolverError < BotoCoreError < Exception
BotoCoreError < Exception
ChecksumError < BotoCoreError < Exception
ClientError < Exception
ConfigNotFound < BotoCoreError < Exception
ConfigParseError < BotoCoreError < Exception
ConnectTimeoutError < [ConnectionError < BotoCoreError < Exception, ConnectTimeout < [ConnectionError < RequestException < OSError < Exception, Timeout < RequestException < OSError < Exception]]
ConnectionClosedError < HTTPClientError < BotoCoreError < Exception
ConnectionError < BotoCoreError < Exception
CredentialRetrievalError < BotoCoreError < Exception
DataNotFoundError < BotoCoreError < Exception
EndpointConnectionError < ConnectionError < BotoCoreError < Exception
EventStreamError < ClientError < Exception
HTTPClientError < BotoCoreError < Exception
ImminentRemovalWarning < Warning < Exception
IncompleteReadError < BotoCoreError < Exception
InfiniteLoopConfigError < InvalidConfigError < BotoCoreError < Exception
InvalidConfigError < BotoCoreError < Exception
InvalidDNSNameError < BotoCoreError < Exception
InvalidExpressionError < BotoCoreError < Exception
InvalidMaxRetryAttemptsError < InvalidRetryConfigurationError < BotoCoreError < Exception
InvalidRetryConfigurationError < BotoCoreError < Exception
InvalidS3AddressingStyleError < BotoCoreError < Exception
MD5UnavailableError < BotoCoreError < Exception
MetadataRetrievalError < BotoCoreError < Exception
MissingParametersError < BotoCoreError < Exception
MissingServiceIdError < UndefinedModelAttributeError < Exception
NoCredentialsError < BotoCoreError < Exception
NoRegionError < BaseEndpointResolverError < BotoCoreError < Exception
OperationNotPageableError < BotoCoreError < Exception
PaginationError < BotoCoreError < Exception
ParamValidationError < BotoCoreError < Exception
PartialCredentialsError < BotoCoreError < Exception
ProfileNotFound < BotoCoreError < Exception
ProxyConnectionError < [ConnectionError < BotoCoreError < Exception, ProxyError < ConnectionError < RequestException < OSError < Exception]
RangeError < ValidationError < BotoCoreError < Exception
ReadTimeoutError < [HTTPClientError < BotoCoreError < Exception, ReadTimeout < Timeout < RequestException < OSError < Exception, ReadTimeoutError < [TimeoutError < HTTPError < Exception, RequestError < PoolError < HTTPError < Exception]]
RefreshWithMFAUnsupportedError < BotoCoreError < Exception
SSLError < [ConnectionError < BotoCoreError < Exception, SSLError < ConnectionError < RequestException < OSError < Exception]
ServiceNotInRegionError < BotoCoreError < Exception
StubAssertionError < [StubResponseError < BotoCoreError < Exception, AssertionError < Exception]
StubResponseError < BotoCoreError < Exception
UnStubbedResponseError < StubResponseError < BotoCoreError < Exception
UndefinedModelAttributeError < Exception
UnknownClientMethodError < BotoCoreError < Exception
UnknownCredentialError < BotoCoreError < Exception
UnknownEndpointError < [BaseEndpointResolverError < BotoCoreError < Exception, ValueError < Exception]
UnknownKeyError < ValidationError < BotoCoreError < Exception
UnknownParameterError < ValidationError < BotoCoreError < Exception
UnknownServiceError < DataNotFoundError < BotoCoreError < Exception
UnknownServiceStyle < BotoCoreError < Exception
UnknownSignatureVersionError < BotoCoreError < Exception
UnseekableStreamError < BotoCoreError < Exception
UnsupportedSignatureVersionError < BotoCoreError < Exception
UnsupportedTLSVersionWarning < Warning < Exception
ValidationError < BotoCoreError < Exception
WaiterConfigError < BotoCoreError < Exception
WaiterError < BotoCoreError < Exception

[swetashre]

@raolivei - Thank you for your post. Can you please try this code and let me know if you are able to handle the error or not ?

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client('s3')
try:
   s3.get_object(Bucket='bucketname',Key='keyname')
except ClientError as e:
   if e.response['Error']['Code'] == 'InvalidObjectState':
           print(e.response['Error'])

Are you getting the error with this code storage_class = s3.Object(BUCKET_NAME, KEY).storage_class ? Can you please give me the exact code you are running with the debug log ? You can enable log by adding boto3.set_stream_logger('') to your code.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[cdibble]

FYI, I had the same issue as the OP, but it was resolved by from botocore.exceptions import ClientError. So it looks like this was not a real boto issue, but just a missing import. Here is my run:

for i in range(0, len(bucket_names)):
    try:
        bucket_objs = client.list_objects_v2(Bucket = bucket_names[i])
        if i == 0:
            s3_dict = {bucket_names[i] : [bucket_objs]}
        else:
            s3_dict.update({bucket_names[i] : [bucket_objs]})
    except ClientError:
        print('Access denied: ' + bucket_names[i])

That runs fine.