Boto3: Add explanation on how to catch boto3 exceptions

[schumannd]

The problem I have with the boto3 documentation can be found here: https://stackoverflow.com/questions/46174385/properly-catch-boto3-errors

Am I doing this right? Or what is best practice when dealing with boto3 exceptions? Can this be added to the wiki?

[stealthycoin]

I'm not sure what wiki page you mean? Do you mean the documentation? Thats not a wiki its generated from the models. So botocore will raise a ClientError which you can always catch for any sort of issue you can then check the code. There is also a newer style of error catching that you can do using the errorfactory which will create an error class based on the codes returned. This was added here. This is not currently documented, we will need to update our documentation generator to add this to the docs. I will mark this as a documentation feature request.

[schumannd]

Yes, I am talking about the errorfactory documentation. Thanks

[Netherdrake]

I had the same question, until I realized that all service exceptions are tucked nicely in .exceptions of the resource/client.

For example, if using iam service:

iam = boto3.client('iam')

try:
    response = iam.create_role(
        Path='/',
        RoleName='Elastic_Transcoder_Viewly_Role',
        AssumeRolePolicyDocument=trust_policy,
        Description='Default Role'
    )
except iam.exceptions.EntityAlreadyExistsException:
    pass

[schumannd]

I use this in the linked SO article. My issue is, that I can't find an overview of what exceptions exist. Having the exceptions in .exceptions of the resource/client is also not ideal when e.g. writing tests as you usually don't have the resource object available there.

[ezramorris]

I'm not sure this always works. I have dynamodb = boto3.resource('dynamodb') but when attempting to except dynamodb.exceptions.ConditionalCheckFailedException I get AttributeError: 'dynamodb.ServiceResource' object has no attribute 'exceptions'.

[ldoran]

@ezramorris You'll need to use boto3.client('dynamodb') to be able to access exceptions, it doesn't work with resources.

[biosopher]

We still need a good explanation on how to catch boto3 errors. E.g. this is a common approach that I've seen but it doesn't work in my case:

try:
            cos_client.download_fileobj(bucket_name, object_key, file)
        except ibm_boto3.exceptions.ibm_botocore.client.ClientError as e:
            print('Error: {}.'.format(e.response['Error']['Message']))
            raise e
        except:
            print(e.__dict__)
            e = sys.exc_info()[0]
            if e.response != None:
                print("Detailed error: ",e.response )

[gyoza]

        from botocore.exceptions import ClientError

        except ClientError as e:
            error_code = e.response['Error']['Code']

in other words, e.response should always contain something.. maybe not always { "Error" { "Message": "xx"} format...

you may want to get rid of

        except ibm_boto3.exceptions.ibm_botocore.client.ClientError as e:
````
and replace with just 

from ibm_boto3.exceptions.ibm_botocore.client import ClientError
except ClientError as e:
```
which may be why you're having issues.

[biosopher]

@gyoza Already got that as shown above. My question is how to get the error message out of ClientError.

[gyoza]

updated.

[biosopher]

Very odd. I just got this error when using your recommended code above:

from ibm_boto3.exceptions.ibm_botocore.client import ClientError
      2 
      3 def download_object_from_cos(cos_client, bucket_name, object_key):
      4     import ibm_boto3
      5     import sys

ImportError: No module named 'ibm_boto3.exceptions.ibm_botocore'; 'ibm_boto3.exceptions' is not a package

[gyoza]

Maybe try removing IBM botocore and boto3 and just use public domain
version... It's hard to say on a forked library..

On Tue, Apr 24, 2018, 9:49 AM Anthony Stevens [e-mail]
wrote:

Very odd. I just got this error when using your recommended code above:

from ibm_boto3.exceptions.ibm_botocore.client import ClientError
2
3 def download_object_from_cos(cos_client, bucket_name, object_key):
4 import ibm_boto3
5 import sys

ImportError: No module named 'ibm_boto3.exceptions.ibm_botocore'; 'ibm_boto3.exceptions' is not a package

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/boto/boto3/issues/1262#issuecomment-384002457, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAibBtgw0bZlwny8xjjuv6KLn-wYBBZCks5tr1eLgaJpZM4PWJwe
.

[samuel-72]

Is there a timeline as to when resource objects would support exceptions

[Lawouach]

Hey there, a friendly ping on this as this is indeed very confusing overall.

[nand0p]

cloudformation client exceptions appear to be incomplete?

['AlreadyExistsException', 'ChangeSetNotFound', 'CreatedButModifiedException', 'InsufficientCapabilitiesException', 'InvalidChangeSetStatus', 'InvalidOperationException', 'LimitExceededException', 'NameAlreadyExistsException', 'OperationIdAlreadyExistsException', 'OperationInProgressException', 'OperationNotFoundException', 'StackInstanceNotFoundException', 'StackSetNotEmptyException', 'StackSetNotFoundException', 'StaleRequestException', 'TokenAlreadyExistsException']

using this method, i cannot seem to catch:

update-stack no-updates-needed:
response['Error']['Code'] == 'ValidationError' and 'No updates are to be performed' in err.response['Error']['Message']
describe-stack does-not-exists:
response['Error']['Code'] == 'ValidationError' and 'does not exist' in err.response['Error']['Message']

except cfn_client.exceptions.StackInstanceNotFoundException does not work for catching the non-existing stack describe

[alexpareto]

An ugly, but workable solution to find out what exceptions are available on each client from the command line:

>>> import boto3
>>> client = boto3.client('sns')  # or whatever client you're using
>>> client.exceptions.__dict__

should print something like this:
{'_code_to_exception': {'AuthorizationError': <class 'botocore.errorfactory.AuthorizationErrorException'>, 'EndpointDisabled': <class 'botocore.errorfactory.EndpointDisabledException'>, 'InternalError': <class 'botocore.errorfactory.InternalErrorException'>, 'InvalidParameter': <class 'botocore.errorfactory.InvalidParameterException'>, 'ParameterValueInvalid': <class 'botocore.errorfactory.InvalidParameterValueException'>, 'NotFound': <class 'botocore.errorfactory.NotFoundException'>, 'PlatformApplicationDisabled': <class 'botocore.errorfactory.PlatformApplicationDisabledException'>, 'SubscriptionLimitExceeded': <class 'botocore.errorfactory.SubscriptionLimitExceededException'>, 'Throttled': <class 'botocore.errorfactory.ThrottledException'>, 'TopicLimitExceeded': <class 'botocore.errorfactory.TopicLimitExceededException'>}}

[kitos9112]

Thanks for this thread guys, I was always importing all exceptions to be used and catch in my classes!

[thirdshade]

I use this in the linked SO article. My issue is, that I can't find an overview of what exceptions exist. Having the exceptions in .exceptions of the resource/client is also not ideal when e.g. writing tests as you usually don't have the resource object available there.

+1 here, I was in lookout for the list of exceptions I can code in my script. Found nothign very useful.
Any documentation or the list of documentation?

[russellballestrini]

Thanks @alexpareto !

import boto3

rds = boto3.client("rds")

rds.exceptions.__dict__['_code_to_exception'].keys()

[webchang]

Regarding catching "ClientError" exception using python package "ibm-cos-sdk", add the following three lines at the beginning:

import ibm_boto3
from ibm_botocore.client import Config
from ibm_botocore.exceptions import ClientError

[faisalburhanudin]

Right now I using something like this

    try:
        dynamodb.create_table(
            TableName=TABLE_NAME,
            KeySchema=[
                {
                    'AttributeName': 'seller_id',
                    'KeyType': 'HASH'
                }
            ],
            AttributeDefinitions=[
                {
                    'AttributeName': 'seller_id',
                    'AttributeType': 'S'
                }
            ],
            ProvisionedThroughput={
                'ReadCapacityUnits': 1,
                'WriteCapacityUnits': 1
            }
        )

    except Exception as e:
        if e.__class__.__name__ == 'ResourceInUseException':
            pass
        else:
            raise e

If anyone have better error handling?

[dacut]

There are currently (as of version 1.12.142) three base exceptions and two warnings:

• BotoCoreError
• ClientError
• UndefinedModelAttributeError
• ImminentRemovalWarning
• UnsupportedTLSVersionWarning

The ClientError exception appears to be raised when a remote call is made but the service returns an error. The BotoCoreError hierarchy appears to reflect issues in creating or handling the call itself. (Why UndefinedModelAttributeError is not a subclass of BotoCoreError is a mystery to me.)

Structure of a ClientError

Within ClientError (but not BotoCoreError), there will be an operation_name attribute (should be a str) and a response attribute (should be a dict). The response attribute should have the following form (example from a malformed ec2.DescribeImages call):

{
    "Error": {
        "Code": "InvalidParameterValue",
        "Message": "The filter 'asdfasdf' is invalid"
    },
    "ResponseMetadata": {
        "RequestId": "aaaabbbb-cccc-dddd-eeee-ffff00001111",
        "HTTPStatusCode": 400,
        "HTTPHeaders": {
            "transfer-encoding": "chunked",
            "date": "Fri, 01 Jan 2100 00:00:00 GMT",
            "connection": "close",
            "server": "AmazonEC2"
        },
        "RetryAttempts": 0
    }
}

Error handling

Typically, you want to handle specific error codes. Because this is largely only quasi-documented, this is what I typically do for error handling (in this case, handling the InvalidParameterValue from my ec2.DescribeImages call):

except ClientError as e:
    error_code = e.response.get("Error", {}).get("Code")
    if error_code == "InvalidParameterValue":
        # Do something
        pass
    else:
        raise

Or, if I need to intermingle it with other exception handling code (much rarer) and am feeling lazy about it:

except Exception as e:
    error_code = getattr(e, "response", {}).get("Error", {}).get("Code")
    ...

Exception hierarchy

Here's the full hierarchy of statically defined exceptions/warnings I found programmatically. Note that some exception classes use multiple inheritance.

AliasConflictParameterError < ValidationError < BotoCoreError < Exception
ApiVersionNotFoundError < BotoCoreError < Exception
BaseEndpointResolverError < BotoCoreError < Exception
BotoCoreError < Exception
ChecksumError < BotoCoreError < Exception
ClientError < Exception
ConfigNotFound < BotoCoreError < Exception
ConfigParseError < BotoCoreError < Exception
ConnectTimeoutError < [ConnectionError < BotoCoreError < Exception, ConnectTimeout < [ConnectionError < RequestException < OSError < Exception, Timeout < RequestException < OSError < Exception]]
ConnectionClosedError < HTTPClientError < BotoCoreError < Exception
ConnectionError < BotoCoreError < Exception
CredentialRetrievalError < BotoCoreError < Exception
DataNotFoundError < BotoCoreError < Exception
EndpointConnectionError < ConnectionError < BotoCoreError < Exception
EventStreamError < ClientError < Exception
HTTPClientError < BotoCoreError < Exception
ImminentRemovalWarning < Warning < Exception
IncompleteReadError < BotoCoreError < Exception
InfiniteLoopConfigError < InvalidConfigError < BotoCoreError < Exception
InvalidConfigError < BotoCoreError < Exception
InvalidDNSNameError < BotoCoreError < Exception
InvalidExpressionError < BotoCoreError < Exception
InvalidMaxRetryAttemptsError < InvalidRetryConfigurationError < BotoCoreError < Exception
InvalidRetryConfigurationError < BotoCoreError < Exception
InvalidS3AddressingStyleError < BotoCoreError < Exception
MD5UnavailableError < BotoCoreError < Exception
MetadataRetrievalError < BotoCoreError < Exception
MissingParametersError < BotoCoreError < Exception
MissingServiceIdError < UndefinedModelAttributeError < Exception
NoCredentialsError < BotoCoreError < Exception
NoRegionError < BaseEndpointResolverError < BotoCoreError < Exception
OperationNotPageableError < BotoCoreError < Exception
PaginationError < BotoCoreError < Exception
ParamValidationError < BotoCoreError < Exception
PartialCredentialsError < BotoCoreError < Exception
ProfileNotFound < BotoCoreError < Exception
ProxyConnectionError < [ConnectionError < BotoCoreError < Exception, ProxyError < ConnectionError < RequestException < OSError < Exception]
RangeError < ValidationError < BotoCoreError < Exception
ReadTimeoutError < [HTTPClientError < BotoCoreError < Exception, ReadTimeout < Timeout < RequestException < OSError < Exception, ReadTimeoutError < [TimeoutError < HTTPError < Exception, RequestError < PoolError < HTTPError < Exception]]
RefreshWithMFAUnsupportedError < BotoCoreError < Exception
SSLError < [ConnectionError < BotoCoreError < Exception, SSLError < ConnectionError < RequestException < OSError < Exception]
ServiceNotInRegionError < BotoCoreError < Exception
StubAssertionError < [StubResponseError < BotoCoreError < Exception, AssertionError < Exception]
StubResponseError < BotoCoreError < Exception
UnStubbedResponseError < StubResponseError < BotoCoreError < Exception
UndefinedModelAttributeError < Exception
UnknownClientMethodError < BotoCoreError < Exception
UnknownCredentialError < BotoCoreError < Exception
UnknownEndpointError < [BaseEndpointResolverError < BotoCoreError < Exception, ValueError < Exception]
UnknownKeyError < ValidationError < BotoCoreError < Exception
UnknownParameterError < ValidationError < BotoCoreError < Exception
UnknownServiceError < DataNotFoundError < BotoCoreError < Exception
UnknownServiceStyle < BotoCoreError < Exception
UnknownSignatureVersionError < BotoCoreError < Exception
UnseekableStreamError < BotoCoreError < Exception
UnsupportedSignatureVersionError < BotoCoreError < Exception
UnsupportedTLSVersionWarning < Warning < Exception
ValidationError < BotoCoreError < Exception
WaiterConfigError < BotoCoreError < Exception
WaiterError < BotoCoreError < Exception

Appendix

Code for how I generated the above (Python 3.7):

def hierarchy(t):
    if t is Exception or t is object:
        return t.__name__

    result = t.__name__ + " < "
    if len(t.__bases__) == 1:
        result += hierarchy(t.__bases__[0])
    else:
        result += "[" + ", ".join([hierarchy(base) for base in t.__bases__]) + "]"
    return result

# Getting the list of base exceptions/warnings
for key, value in sorted(botocore.exceptions.__dict__.items()):
    if isinstance(value, type) and (value.__bases__ == (Exception,) or value.__bases__ == (Warning,)):
        print(key)

# Printing the hierarchy
for key, value in sorted(botocore.exceptions.__dict__.items()):
     if isinstance(value, type):
        print(hierarchy(value))

[schumannd]

@dacut I don't think your list is complete. At a glance I could not find the NotFoundException (see the comment of @alexpareto here

[dacut]

@schumannd The NotFoundException is a factory-generated exception (created when the client in question is imported within botocore.errorfactory) and won't show up in that list. I've traditionally not used the factory-generated exceptions because they've wreaked havoc with analysis tools like mypy; however, it looks like the current version of mypy handles this (somewhat convoluted) code fine:

import boto3

sns = boto3.client("sns")

class App(object):
    def __init__(self, sns):
        super(App, self).__init__()
        self.sns = sns

    def hello(self):
        try:
            self.sns.list_subscriptions_by_topic(TopicArn="asdfasdf")
        except self.sns.exceptions.InvalidParameterException as e:
            print(f"IPE: {e}")

app = App(sns)
app.hello()

I'm still not sure how I feel about using those types, but seeing that mypy swallowed this without a complaint was a happy discovery today.

Edited my earlier comment to clarify that the hierarchy is the statically defined exceptions only.

[danizen]

It would be nice if ExpiredTokenException were a named exception, or if I knew how to generate a class based on it:

-> LOG.exception("ssm:GetParameter(Name='/standard/pAwsAccount')")
(Pdb) p e
ClientError('An error occurred (ExpiredTokenException) when calling the GetParameter operation: The security token included in the request is expired',)
(Pdb) dir(e)
['MSG_TEMPLATE', '__cause__', '__class__', '__context__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__suppress_context__', '__traceback__', '__weakref__', '_get_retry_info', 'args', 'operation_name', 'response', 'with_traceback']
(Pdb) p e.operation_name
'GetParameter'
(Pdb) p e.response['Error']['Code']
'ExpiredTokenException'
(Pdb) e.__class__.__name__
'ClientError'

[swetashre]

Following up on old issue. This document contains all details about how to handle boto3 exceptions
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html

@schumannd - Does this solve your problem ?

[schumannd]

@swetashre I think it summarizes the situation well.

I tested it by looking for the two exceptions from my SO post.

One of them I couldn't find (botocore.errorfactory.NotFoundException), but maybe I didn't look long enough.

The other one I found in this file:
https://github.com/boto/boto/blob/master/boto/cognito/identity/exceptions.py

But it was not listed under statically defined exceptions, and not listed in the AWS docs as far as I could tell.

It seems like those errors should be listed in the "statically defined exceptions" list in the article. Or it should be noted, that the list is not complete.

So it is a good step in clearing up the boto core exceptions, but not perfect.

[eastside]

Here's an idea:

What if the boto3 library itself just reported an _importable_ exception when raising an error, instead of something that can not be imported?

Like instead of botocore.errorfactory.ResourceNotFoundException it raises botocore.exceptions.ClientError? What's the point of the errorfactory anyway? I don't think it serves ANY point if you can't import the exception and do something with it.

[bitsofinfo]

experiencing the same, can't seem to catch this botocore.errorfactory.QueueDoesNotExist: An error occurred (AWS.SimpleQueueService.NonExistentQueue)