Boto3: KeyError: 'endpoint_resolver'

[vc-utelly]

Hi,

I sometimes get that error trying to call a lambda function
boto3==1.3.1

def lambda_execute(payload):
    import boto3
    client = boto3.client('lambda', aws_access_key_id=KEY, aws_secret_access_key=SECRET region_name=REGION)
    client.invoke(**payload)

payload is in this format:

{'FunctionName': fct, 'InvocationType': 'Event', 'LogType': 'None', 'Payload': simplejson.dumps(payload, default=encode_model)}

error seems to be coming from get_component in botocore/session.py

Can you help ?

[jamesls]

Is this function being called in multiple threads? The only way I see that happening is if boto3.client is being invoked in multiple threads (client creation is not thread safe).

[vc-utelly]

Hi James,
Yes it is - weird it hasn't been raised before.
would you recommend this as a solution? https://geekpete.com/blog/multithreading-boto3/

[JordonPhillips]

@vinczente Yes, each thread should have its own session.

[cpury]

I'm getting this in one case out of tens of thousands, but consistently in this case. I'm already creating a resource per thread... My code inside the thread is:

awslambda = boto3.client('lambda')

This happens inside a lambda function (it invokes other lambda functions). The exact error is:

Error in thread: Traceback (most recent call last):
  File "/var/task/service.py", line 78, in call_recursively
    awslambda = boto3.client('lambda')
  File "/var/runtime/boto3/__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "/var/runtime/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/var/runtime/botocore/session.py", line 826, in create_client
    endpoint_resolver = self.get_component('endpoint_resolver')
  File "/var/runtime/botocore/session.py", line 701, in get_component
    return self._components.get_component(name)
  File "/var/runtime/botocore/session.py", line 901, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'

I tried replacing it with

session = boto3.session.Session()
awslambda = session.resource('lambda')

But this causes:

Error in thread (p. 10-18): Traceback (most recent call last):
  File "/var/task/service.py", line 79, in call_recursively
    awslambda = session.resource('lambda')
  File "/var/runtime/boto3/session.py", line 347, in resource
    has_low_level_client)
ResourceNotExistsError: The 'lambda' resource does not exist.
The available resources are:
   - cloudformation
   - cloudwatch
   - dynamodb
   - ec2
   - glacier
   - iam
   - opsworks
   - s3
   - sns
   - sqs

Consider using a boto3.client('lambda') instead of a resource for 'lambda'

Any other options to try?

[cpury]

So it seems you can only create lambda clients using the default session? I'm not allowed to get it from a custom session, and boto3.client(...) always uses the default session. Why?

[vc-utelly]

@cpury Looking back at my code, I cannot remember why I didn't go with the session solution (perf? maybe) but what I have now is (simplified):

lambda_client = boto3.client('lambda', aws_access_key_id='xxx', aws_secret_access_key='xxx', region_name='eu-west-1')

def execute_async(fct, payload):
        threading.Thread(target=lambda_execute({'FunctionName': fct, 'InvocationType': 'Event', 'LogType': 'None', 'Payload': simplejson.dumps(payload, default=encode_model)}))

def lambda_execute(payload):
    lambda_client.invoke(**payload)

then the code is calling execute_async
Hope that helps ?

[cpury]

@vinczente thanks for your code! So you're using a shared client for all threads?

My code is similar, but more complex. I'm using a ThreadPool, though. And I create the client in each thread.

I should have emphasized more clearly: My code works in 99.9999% of all cases! The lambda function (that calls other lambda functions) has been run on tens of thousands of files as inputs, and never failed. There is exactly one file where this error consistently appears.

I assume there are weird edge cases where this can happen, and you might run into it yourself at some point :(

So it would be great if at least there was a way to create a lambda client that's not using the default session, as that would likely fix this.

[cpury]

Wow, super weird! Inspired by your code, I changed mine to use a global lambda client object that's shared among all threads. AKA, do exactly the opposite of what's recommended here in this ticket. And that fixes it!

So for anyone running into the same issue: Just use a global lambda object. And if anyone of the boto3 team wants to explain how this makes sense, please do, I'm curious :)

[khuevu]

@cpury , I observed the same issue too. It gave the exception when I use separate client per thread and only works if I use one global client.

[junghoon-picwell]

@cpury: I ran into the same issue as well. I wasn't comfortable creating a global client due to this documentation, so I tried until I get a client, e.g.

    client = None
    while not client:
        try:
            client = boto3.client('lambda')
        except:
            client = None

This seems to work.

[cpury]

@junghoon-picwell that looks scary, but great that it works! I've changed jobs so can't confirm if that fixes our case.

[junghoon-picwell]

Yes, it does make me uncomfortable. I introduced retry limits now, but I don't think it is still good. Any thoughts?

[claweyenuk]

Is this issue fixed after #73?

I hit this exception, and thought threading might be the issue, but it might have been a false alarm. It turned out the proxy server I was using went down. When I switched to a working proxy server, the KeyError: 'endpoint_resolver' went away.

I was using threads, and I haven't yet had a chance to dig into whether pointing to a bad proxy server while not using threading triggers this.

[rkiyanchuk]

I observe similar behavior on v1.9.16, but the error is KeyError: 'credential_provider' and it depends on the number of workers in a ThreadPool. For me the error seems to happen only when the number of threads is more than half (n/2+1) of tasks.

Making client global also "fixes" the error.

1592 seems to be relevant.

[dineshmc2]

Ideal fix would be

session = boto3.session.Session()
awslambda = session.client('lambda')

(note that its not using sesion.resource() which is only applicable for small number of resources)

All the clients created using boto3.client()/boto3.resource() share default global session and thus aren't thread safe.
https://github.com/boto/boto3/blob/c9399569448890ea78f5ba8e8ae8a55b8fddc3ce/boto3/__init__.py#L85-L100

[matse004]

The problem with the "global fix" for me is that with the shared lambda client I get about 50% more invocations than I have workers/threads.

Can't explain why that's happening, but a client per session fixes that. When it works :) And it often doesn't.

[hashark]

It is a relatively old issue but maybe that will help someone who bumps into it now:

I used a solution similar to the one proposed in https://github.com/boto/boto3/pull/806 (I used threading.RLock before creating a client)
I tried the solution of @dineshmc2 (creating new session and new client every time) but it was not as efficient (time wise) as the solution with the locks (I tried creating 100 clients using threads. Creating new session for each thread took about 1 minute while locking took less than 10 seconds)

I'll be happy to learn of different solutions if someone has.

[arnav13081994]

I am uploading media files client side and was getting the error: KeyError: 'endpoint_resolver'

Somehow making s3.client global fixed the issue. I have no idea why that happened.

Just putting this here for anyone who is facing the same issue.

[JarnoRFB]

I got the same issue when copying files in a thread pool. What helped me was the answer suggested here https://stackoverflow.com/a/57445139/4873972. Basically switching to a ProcessPool and initializing each process with its own client.

[shranith]

A simple solution based on what I read above, if you want to use threads and not multiprocessing, - works fine.

cond = True
while cond:
    try:
        s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
        cond = False
    except:
        cond = True

[puzzle150]

A simple solution based on what I read above, if you want to use threads and not multiprocessing, - works fine.

cond = True
while cond:
    try:
        s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
        cond = False
    except:
        cond = True

Thanks for your answer, it works for me :)

[ali1366]

I changed s3 = boto3.resource('s3') to below and it work for me
session = boto3.session.Session() s3 = session.resource('s3')

[swetashre]

Going over old issue. Is anyone still having the issue ? If anyone has any concerns please reopen a new issue, i would be happy to help.

[jsollom]

@swetashre , I have run into this same issue. I have a multi-threaded application where each thread creates its own boto3 client.

Running with:
python==3.8.2
boto3==1.12.9
botocore==1.15.9

Code snippet from the thread code:
{code}
s3 = boto3.client('s3',
endpoint_url= s3_protocol + "://" + s3_gateway,
aws_access_key_id= s3_access_key,
aws_secret_access_key= s3_secret_key,
use_ssl=False,
verify=False,
config=BotoConfig(
connect_timeout=connection_timeout,
read_timeout=read_timeout))
{code}

I still encounter this error. I feel that because this error still exists in current code bases that this bug should not be closed. It contains good information regarding the issue.

[farrellit]

We've run into this recently with python 2.7 and bot 1.10.16.

We think we likely need to work around it with uwsgi lazy loading setting toggle. but still looking through it.

[golharam]

I'm running into this now. I have a Flask application sitting behind Apache. When multiple clients are accessing a resource in the Flask app, that request triggers a boto3.client('s3').list_objects call, where this error comes up.
My app is using "boto3":"1.7.4" and "botocore":"1.10.6".

[cmcmillon]

Try using concurrent.futures