Botocore: ValueError: Unknown component: endpoint_resolver

[russellneufeld]

I'm seeing a very sporadic error which I'm hoping someone here can explain. See stack trace below. This is happening less than 5 times a day on a site that's getting maybe a million requests a day, so it's pretty rare, but I'd like to understand what's going on and see if there's a way to fix it. I should also mention that the EC2 instance this code runs on has an IAM role associated with it, and gets its creds automatically through that mechanism.

Any idea what might be going wrong here? Thanks.

$ pip freeze | grep boto
boto3==0.0.19
botocore==1.0.0b1

File "/home/ubuntu/nasuni/nasuni/utils/aws/s3.py", line 82, in list_bucket_contents
bucket = boto3.resource('s3', region_name=settings.AWS_REGION).Bucket(bucket)
File "/usr/local/lib/python2.7/dist-packages/boto3/init.py", line 87, in resource
return _get_default_session().resource(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/boto3/session.py", line 270, in resource
aws_session_token=aws_session_token, config=config)
File "/usr/local/lib/python2.7/dist-packages/boto3/session.py", line 192, in client
aws_session_token=aws_session_token, config=config)
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 833, in create_client
endpoint_resolver = self.get_component('endpoint_resolver')
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 741, in get_component
return self._components.get_component(name)
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 857, in get_component
raise ValueError("Unknown component: %s" % name)
ValueError: Unknown component: endpoint_resolver

[jamesls]

Could you tell us a little more about what the code is doing? Is this being run in a multithreaded context? Could you share any additional code that demonstrates this issue?

[russellneufeld]

Yes, it's running within mod_wsgi under Apache in a multi-threaded context.

In high traffic situations with the boto library (not boto3) I saw problems where every so often attempts to query the metadata service to get the IAM role assigned to an EC2 instance would fail, causing very weird errors. This kind of feels like the same thing.

[jamesls]

Based on the traceback, it looks like you're creating a session and then creating a bucket. One thing worth pointing out is that while clients are thread-safe, we recommend either using a session per thread, or having the main thread create a session and have a resource per thread (http://boto3.readthedocs.org/en/latest/guide/resources.html?highlight=threaded#multithreading). The Session object, particularly it's initialization, isn't currently thread safe.

[russellneufeld]

@jamesls - thanks for the update and sorry for my slow response. I don't think I'm creating a session and then creating a bucket, but perhaps I'm wrong. Here's the line of code from the traceback which occasionally causes an error:

bucket = boto3.resource('s3', region_name=settings.AWS_REGION).Bucket(bucket)

This is running within mod_wsgi in daemon mode, so it's within a multi-threaded environment, however this code runs within the local context of a thread.

...

Actually, I just read the boto3 code and it looks like within the boto3 library it's calling _get_default_session(), which is definitely going to be accessed by multiple threads simultaneously and thus hit the the thread-safety problems you are talking about. Looks like I'll need to change my code to something where I create a Session object explicitly and then use it, rather than the API which I was using which creates the Session automatically.

Thanks for the pointer in the right direction.

Russ

[jamesls]

Sounds good. Let us know if you run into any other issues.

[russellneufeld]

For what it's worth, I think this is the code with the thread safety issue (from botocore/session.py):

    def get_component(self, name):
        if name in self._deferred:
            factory = self._deferred.pop(name)
            self._components[name] = factory()
        try:
            return self._components[name]
        except KeyError:
            raise ValueError("Unknown component: %s" % name)

If two threads evaluate the "name in" check to True, only one will successfully pop on the next line. The other will fail. Seems like putting a lock around this section would fix things.

In the meantime I'm putting this in my startup code under a lock:

boto3.setup_default_session()
boto3.DEFAULT_SESSION._session.get_component('data_loader')
boto3.DEFAULT_SESSION._session.get_component('event_emitter')
boto3.DEFAULT_SESSION._session.get_component('endpoint_resolver')
boto3.DEFAULT_SESSION._session.get_component('credential_provider')

[jdolitsky]

I ran into this and @russellneufeld 's "meantime" solution worked for me. Thank you. I was having the issue when making requests in parallel using asyncio

[thehesiod]

btw, even with @russellneufeld 's fix, I ran into huge memory consumption with ~100 threads when running under linux with python 3.4.2 (OSX was OK, ~250MB. I since switched to aio-s3 and it runs really well with 300+ async workers with ~50MB footprint. Our enhancements to that are in https://github.com/tailhook/aio-s3/pull/9. I've since been contemplating switching to https://github.com/jettify/aiobotocore

[komuw]

This is for my future self who will come looking at this issue, again:
Wrap boto3 initialization in a thread lock[1]

1. https://github.com/getsentry/sentry/commit/f58ca5eef79a8fc1091703d72611e8e3e63b34a0