Boto3: It is impossible to catch a CrawlerRunningException

[kdbanman]

This code is used to trigger a crawler:

def trigger_glue_crawler(access_key_id, access_key_secret):
    glue_client = boto3.client(
      "glue",
      aws_access_key_id=access_key_id,
      aws_secret_access_key=access_key_secret
    )
    response = glue_client.start_crawler(Name="example_crawler")
    print("Schema crawler started.")

But if the crawler is already running, this exception is thrown:

Traceback (most recent call last):
  File "example.py", line 102, in trigger_glue_crawler
    response = glue_client.start_crawler(Name="example_crawler")
  File "/lib/python3.6/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/lib/python3.6/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.CrawlerRunningException: An error occurred (CrawlerRunningException) when calling the StartCrawler operation: Crawler with name example_crawler has already started

Trying to catch the exception looks like this:

def trigger_glue_crawler(access_key_id, access_key_secret):
  try:
    glue_client = boto3.client(
      "glue",
      aws_access_key_id=access_key_id,
      aws_secret_access_key=access_key_secret
    )
    response = glue_client.start_crawler(Name="example_crawler")
    print("Schema crawler started.")
  except botocore.errorfactory.CrawlerRunningException:
    print("Schema crawler already running.")

But that exception doesn't exist:

Traceback (most recent call last):
  File "example.py", line 121, in <module>
    trigger_glue_crawler(args.access_key_id, args.access_key_secret)
  File "example.py", line 105, in trigger_glue_crawler
    except botocore.errorfactory.CrawlerRunningException:
AttributeError: module 'botocore.errorfactory' has no attribute 'CrawlerRunningException'

That's not super surprising - I'm guessing that the exception class is dynamically generated based on a server response. But it doesn't exist in boto3.exceptions either. This snippet:

def trigger_glue_crawler(access_key_id, access_key_secret):
  try:
    glue_client = boto3.client(
      "glue",
      aws_access_key_id=access_key_id,
      aws_secret_access_key=access_key_secret
    )
    response = glue_client.start_crawler(Name="example_crawler")
    print("Schema crawler started.")
  except boto3.exceptions.CrawlerRunningException:
    print("Schema crawler already running.")

results in the following error:

Traceback (most recent call last):
  File "example.py", line 120, in <module>
    trigger_glue_crawler(args.access_key_id, args.access_key_secret)
  File "example.py", line 104, in trigger_glue_crawler
    except boto3.exceptions.CrawlerRunningException:
AttributeError: module 'boto3.exceptions' has no attribute 'CrawlerRunningException'

As far as I can tell, it is impossible to specifically catch this exception. But it's a pretty normal flow that I would like to tolerate with a different code path than other exceptions.

[JordonPhillips]

Generally there are two ways for you to catch exceptions. The first is to just use ClientError:

try:
    client.start_crawler(Name="example")
except ClientError as e:
    if e.response.get('Error', {}).get('Code') == 'CrawlerRunningException':
        print('already running')
    else:
        raise

The second is to use the generated exceptions that are attached to the client:

try:
    client.start_crawler(Name="example")
except client.exceptions.CrawlerRunningException:
    print('already running')

The reason it's a bit confusing is that those classes are generated at runtime rather than being a bit of code that's always there.

[kdbanman]

@JordonPhillips The second mechanism is exactly what I'm after. I didn't realize the dynamically generated exception classes would be attached to the client.

Is there a place I can look for documentation on that mechanism? I would like to understand where else it is applied.

[JordonPhillips]

The second bit isn't really documented anywhere because it's not 100% complete. You see our clients are generated by service models, and there's some exceptions that aren't defined there. We're still working out how we want to support non-modeled exceptions: boto/botocore#1489

[kdbanman]

Thanks again @JordonPhillips! Very helpful - good luck with the documentation :)