Boto3: ClientError feels very general

[aidos]

I've just started using boto3 (after years with the previous version) so I've been getting to grips with the new api today.

One thing that stuck out at me is how one needs to catch and inspect ClientError to determine existence when working at the resource level (eg https://boto3.readthedocs.org/en/latest/guide/migrations3.html#accessing-a-bucket).

Maybe I'm missing something but it feels like those various exceptions should be wrapped as more specific exceptions so they could at least be caught directly. Something along the lines of:

try:
    s3.meta.client.head_bucket(Bucket='mybucket')
except BucketNotFoundError as e:
    pass

As it stands it seems I need to do a bunch of 404 checks for testing if s3 keys exist etc. Maybe there's a better way of doing it?

It does feel a little like the level of abstraction is a bit close to the underlying aws implementation - but maybe that's the intention?

As ever, thanks for an amazing and solid bit of software.

[wayne-luminal]

+1. I've also found error handling to be a little cumbersome in this regard. Are named exceptions on the roadmap?

Great work on getting boto3 together!

[jamesls]

I think we can improve on this. Just brainstorming an idea here:

We have to keep in mind that the methods available on resources/clients are generated at runtime, which would mean that the exceptions would have to be generated at runtime as well. One thing we could do is attach this to the client objects (and similarly for resources). It could look something like this:

try:
    client.create_user(...)
except client.exceptions.EntityAlreadyExists:
    handle_error()
except client.exceptions.NoSuchEntity:
    handle_other_error()
...

It also has the benefit of not requiring having to from botocore.exceptions import ClientError in boto3 code.

Thoughts?

[wayne-luminal]

@jamesls I like the look of this. One question, how do you see exceptions being handled with dots in the code? For example, InvalidAMIID.NotFound. To preserve the dot notation we would need a hierarchy of attributes. Or it could be replaced with an underscore? Just spitballing a little here.

[kislyuk]

+1. I would prefer if an error class were vivified from ClientError.response["error"]["code"], so instead of writing:

try:
    ...
except botocore.exceptions.ClientError as e:
    if e.response.get("Error", {}).get("Code") == "...":
        ...
    else:
        raise

or something else equally brittle and yucky, I could just write except botocore.exceptions.LimitExceeded, etc. If they all inherit from ClientError, this is a fully backward compatible change.

[doublerr]

:+1:

[shileiyu]

Is it possible to define exceptions by json data file, just like a resource, so that we could clearly know what exception should be awarded.

[rotsee]

+1

[bilalahmad99]

+1, is this on the roadmap?

[mr337]

+1

[lanceblais]

+1

[yeukhon]

+1 it feels cumbersome to parse the exception...very unpython like.

[HaraldNordgren]

On second though, Kislyuk's pull request (https://github.com/boto/botocore/pull/1005) is probably an overall better solution for all kinds of ClientErrors.

But the project maintainers are seriously stalling on the PR. So if feel like you need this feature now then maybe post a comment there to light a fire under their asses. :grinning:

[jseakle]

Any news on when this is coming? Would be very nice.

[stealthycoin]

This guy was already merged in: boto/botocore#1113. Does that do what you need?

[jseakle]

oh hey, it does! Thanks very much.

[sidewinder12s]

@stealthycoin Are there any docs/updated docs for how to handle errors with the new structure? The only general thing I can find is this: http://botocore.readthedocs.io/en/latest/client_upgrades.html#error-handling, which I am not sure is still completely applicable.

[HaraldNordgren]

I don't think it has been officially released. It's just in the 'develop' branch at the moment, isn't it? So it makes sense that the docs wouldn't be updated.

[sidewinder12s]

@HaraldNordgren Looks like it was released in 1.5?: https://github.com/boto/botocore/blob/master/CHANGELOG.rst#150

[pas256]

@garnaat Get in here and fix this once and for all
:)

[markddavidoff]

If anyone else ends up here looking for a list of error codes, the ClientError.response['Error']['Code'] error codes come from the aws sdk and can be found here

[swetashre]

Anyone looking for documentation related to boto3 error handling
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html

[swetashre]

It looks like this issue has been fixed with the merging of PR https://github.com/boto/botocore/pull/1113
Please let us know if anyone has any more concerns.