Boto3: Exceptions coming from boto3/botocore when running boto3.client('sts') too many times simultaneously

[moalexmonk]

import boto3, threading
for i in range(50):
    threading.Thread(target=lambda: boto3.client('sts')).start()

And you get, tested on my Windows 10 machine (boto3 version 1.5.31, botocore version 1.8.45) and also on an Amazon Linux EC2, a bunch of exceptions like this:

Exception in thread Thread-20:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "<stdin>", line 2, in <lambda>
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 850, in create_client
    credentials = self.get_credentials()
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 474, in get_credentials
    'credential_provider').load_credentials()
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
    del self._deferred[name]
KeyError: 'credential_provider'

or:

Exception in thread Thread-24:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\Python35-32\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "<stdin>", line 2, in <lambda>
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\boto3\session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 851, in create_client
    endpoint_resolver = self.get_component('endpoint_resolver')
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 726, in get_component
    return self._components.get_component(name)
  File "C:\Users\alexander.monk\AppData\Roaming\Python\Python35\site-packages\botocore\session.py", line 926, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'

Normally seems to be several credential_provider ones followed by several endpoint_resolver ones.
The chance of getting these exceptions seems to increase with the number of threads.

[moalexmonk]

Workaround for this in the mean time: Put a boto3.client('sts') call immediately before the for loop.

[jamesls]

You're correct about this behavior. Historically we've said that session methods aren't thread safe, but once you've created a client or resource, we guarantee that those calls are thread safe (http://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing).

While we can investigate changing that stance, I suspect there will be more work than just the component logic, though that's a good start and one of the most common content points in sessions.

[moalexmonk]

So it looks like another way around this is boto3.Session().client (or boto3.session.Session() as in that linked page) instead of just boto3.client.

[zgoda-mobica]

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I'm on safe side but I keep finding statements contradicting the docs.

[rkiyanchuk]

There is also an outstanding unanswered question in botocore about thread safety. Would be great to know what's the actual correct approach should be.

[johnnytardin]

I'm having the same problem. I'm calling boto3.client('sts') for each thread created and i hit KeyError: 'endpoint_resolver'.

[azec-pdx]

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I'm on safe side but I keep finding statements contradicting the docs.

@zgoda-mobica , Same issue here. I had ThreadPoolExecutor and each thread was invoking boto3.client('lambda').invoke() without using Session() which throws KeyError: 'credential_provider' error.

So now using following in thread invoked method....

session = boto3.session.Session()
lambda_client = session.client('lambda')
lambda_response = lambda_client.invoke(...)
....

and it seems to be ok.

[rohnigam]

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I'm on safe side but I keep finding statements contradicting the docs.

@zgoda-mobica , Same issue here. I had ThreadPoolExecutor and each thread was invoking boto3.client('lambda').invoke() without using Session() which throws KeyError: 'credential_provider' error.

So now using following in thread invoked method....

session = boto3.session.Session()
lambda_client = session.client('lambda')
lambda_response = lambda_client.invoke(...)
....

and it seems to be ok.

This solves it, but makes the overall threaded push a bit slow.

[hannes-ucsc]

The documentation says that it is "recommended" to create a resource per thread but the example code below that recommendation creates a resource AND a session per thread. IOW, it is ambiguous if sessions can be shared between threads or not. Further down, the documentation states that resources are NOT thread-safe. That would imply that it's not merely "recommended" to create a resource per thread, but rather that it's a requirement to do so.

I'm confused.

[andrewdunai]

The documentation says that it is "recommended" to create a resource per thread but the example code below that recommendation creates a resource AND a session per thread. IOW, it is ambiguous if sessions can be shared between threads or not. Further down, the documentation states that resources are NOT thread-safe. That would imply that it's not merely "recommended" to create a resource per thread, but rather that it's a _requirement_ to do so.

I'm confused.

I second this. There doesn't seem to be a generic best practice for this case.