Botocore: Connection pool is full, discarding connection

[code-tree]

I'm getting a massive amount of these exceptions while trying to retrieve S3 logs, as I'm retrieving each object in separate threads.

WARNING:botocore.vendored.requests.packages.urllib3.connectionpool:Connection pool is full, discarding connection: bucket.s3.amazonaws.com

Should I be closing connections after using them? Or is this in boto's hands?

thanks

[jamesls]

Should not have to manually close any connections, botocore should be handling this for you. If you have sample code the demonstrates the issue I'd be happy to investigate further.

[code-tree]

Hi James, thanks for helping

I'm trying to retrieve (and then delete) all logs present in an S3 bucket, in the fastest possible way. I'm retrieving them all in separate threads, so that it doesn't wait for one request to complete before starting another. I wrote the code in a bit of a rush, and just now noticed that all threads are using the same client object, which might be the reason for the above exceptions.

I'm not aware of how botocore internals work. Does a client object reuse connections? / Avoid repeat TLS setup?

What is the most efficient way to fetch multiple objects?

thanks,
Jon

def fetch_logs(bucket):
    s3 = boto3.client('s3')

    # Retrieve list of object keys for logs
    objects = s3.list_objects(Bucket=bucket, Prefix='logs/')
    if 'Contents' not in objects:
        return
    objects = [obj['Key'] for obj in objects['Contents']]

    # Start processing objects
    threads = []
    for key in objects:
        threads.append(Thread(target=_fetch_log, args=[s3, bucket, key]))
        threads[-1].start()

    # Wait till all objects processed
    for thread in threads:
        thread.join()


def _fetch_log(s3, bucket, key):
    body = s3.get_object(Bucket=bucket, Key=key)['Body'].read().decode('utf-8')
    process...
    s3.delete_object(Bucket=bucket, Key=key)

[JordonPhillips]

I'm having trouble reproducing this issue. Using the code as you have it, I don't run into it. I modified it to run on more than 1000 objects at a time and ran into a RuntimeError rather than a connection pool. I'm gonna try with bigger objects to see if that does anything, but in the meantime could tell me approximately the size and number of objects you're looking at?

Also, just to note, list_objects returns a max of 1000 objects at a time.

[JordonPhillips]

This issue is being closed due to inactivity

[codemang]

I'm having the same issue when writing to DynamoDB using 20 connections/threads. Any suggestions on debugging/fixing it?

[code-tree]

Not sure, sorry, haven't had the time to investigate further. Though it doesn't seem to break anything for my app. I wonder if it is affecting network efficiency though.

[codemang]

Yes the network calls complete eventually but I'm experiencing delays of up to 10 minutes for a program that should take ~1 minute.

[odinplus]

Got this on my experiments with Celery + eventlet + DynamoDB.
python 2.7.9
boto3 (1.2.3)
botocore (1.3.26)
celery (3.1.20)
eventlet (0.18.3)

Celery worker code:
session = boto3.session.Session()
dynamodb = session.resource('dynamodb')
table = dynamodb.Table('users')

@app.task(bind=True)
def add_user_link(self, i):
try:
result = table.put_item(
Item=i,
ConditionExpression = Attr('uid').not_exists()
)
except ClientError as e:
#print e.message
#print(i)
pass
return True

Got that error when I run this worker with more when 10 threads like this (-c parameter): celery worker -A dynamo -P eventlet -c 20

Found 10 is hardcoded in /usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/adapters.py :
DEFAULT_POOLSIZE = 10
...
def init(self, pool_connections=DEFAULT_POOLSIZE,
pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
Changed 'pool_maxsize=DEFAULT_POOLSIZE' to 'pool_maxsize=100' and can run it with up to 100 threads.

[synhershko]

@JordonPhillips this happens for me as well with boto3, I can share the code that's causing it while trying to download several hundred of files from S3 - or is this a known issue now?

[synhershko]

Please also see related discussion: https://github.com/boto/boto3/issues/268#issuecomment-199759246

[christophlingg]

This one can happen if the number of threads that create a http connection is higher than the default connection pool size of botocore. which is 10.

https://github.com/boto/botocore/issues/766

[farshidz]

Is the connection pool shared between sessions? I'm creating a session per thread (because resources and clients are not thread safe, see http://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading). But I'm still getting this message

[srikiraju]

Wondering the same thing, as @farshidz is this connection pool a process singleton? It's not super obvious to me what the lifetime of the pool is

[oscargws]

I'm having the same issue when trying to send bulk emails out with SES

[JustinTArthur]

Same with Lambda.

[oscargws]

Not sure what your use case is @JustinTArthur, but I was able to fix this problem in my situation. I was attempting to send requests to SES with a Lambda and after about 1000ish emails sent on different threads I'd get the connection pool error. For some reason, new threads kept being made and weren't being discarded.

What fixed it was changing how I instantiated my ThreadPoolExecutor from this:
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:

to this:
with concurrent.futures.ThreadPoolExecutor(4) as executor:

And also making sure that I wasn't going over my maximum send rate for AWS.

[JustinTArthur]

Thanks, @oscargws. In my case, it's multiple gevent greenlets sharing the same Lambda API connection pool instead of threads, but similar story.

I've taken a look at the botocore code re @farshidz and @srikiraju's query and it looks like the urllib3 connection pool is instantiated during session.create_client(), so a pool of 10 connections per session client.

[matse004]

Same with Lambda.

Same with SQS

[JustinTArthur]

This warning is fine. If you dig into the urllib3 connection pool code, it's basically the pool of persistent connections and not the maximum number of concurrent connections you can have. The connections in the pool are re-used if they haven't been closed by the endpoint.

If you'd like to boost the size of this pool, it can be done per-endpoint using the low-level botocore config. boto3 example:

import boto3
import botocore

client_config = botocore.config.Config(
    max_pool_connections=25,
)
boto3.client('lambda', config=client_config)

Botocore Config Documentation:
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

[myselfdesai]

Solution here .. use new feature max_pool_connections
s3_client = boto3.client('s3', config=botocore.client.Config(max_pool_connections=50))

[JuanAntonio03]

I am having the same issue with a package that use boto3 for reading parquets when they are in S3.
Why doesn't the max pool connections parameter appear in the docs so I can set it from a config file or environment variable?