Botocore: Support for customize the max connections and max pools

[pfreixes]

When Boto is used to perform multiple requests concurrently, for example to sped up the write throughput for DynamoDB, it uses the default parameters given by the HTTPAdapter [1], that is currently set to 10 by default.

This number means all of those amount of connections over this number will be dropped and they will not be recycled after, therefore when a new connection is necessary - and the first 10 are bussy - a full handshake TCP trip is performed, that will add a important latency for the whole time operation.

The idea will be add the properly fields to customize the current max connections and max pools, giving to the user the freedom to set this values to other ones different than the default ones.

Does it make sense for you ? if it does, I can try to send the properly pull request with a tentative solution.

[1] https://github.com/boto/botocore/blob/develop/botocore/vendored/requests/adapters.py#L82

[mtdowling]

Thanks for the feature request. I too think this is something we should look at adding either by exposing new parameters or by allowing customers to inject in custom adapters.

[pfreixes]

Hi @mtdowling have a look to the following PR, if there is something that you miss ping me! Also I can handle the Boto3 side once it is merged.

Cheers

[wiltzius]

This is affecting our organization as well so I would love to have a way to manually adjust the connection pool size. We're occasionally exceeding the limit because our multi-threaded web server can sometimes make > 10 requests to S3 in parallel.

When this happens we start getting "Connection pool is full, discarding connection" errors (from urllib3, which must be used under the hood by boto). I believe everything continues to function fine, because we aren't hitting any hard limit merely overflowing the pool, but our application would be more efficient (and less noisy) if we could adjust the pool size.

I'd be open to any suggested workarounds, too. The only suggestion I can find for altering the underlying connection pool size when you don't control the calling code is monkey-patching the urllib3 pool manager, as described here: http://stackoverflow.com/questions/18466079/can-i-change-the-connection-pool-size-for-pythons-requests-module

... but that seems a little extreme.

Thanks!

[alcy]

Similar thoughts as @wiltzius. Everything seems to work fine, but it would be nice to have this feature.

[ryan-lane]

Are there any updates here? This causes me a lot of heartache in production.

[jamesls]

Looking at the original PR, I think I agree with @kyleknap's option of adding a max_concurrency or max_pool_size client config option would be the best thing to add.

I want to make sure I'm caught up on this issue. The main issue people are running into is that when the pool size is reached, we have to establish a new TCP connection and doing the full handshake is causing undesirable latency when using certain services (particularly dynamodb).

Are there any other issues people are running into in production besides that? It would help us get a better understanding of whether or not adding max_pool_size/max_conns would be sufficient, or whether there's more we'd need to do.

[pfreixes]

fair enough.

[wiltzius]

@jamesls your understanding of the issue is correct. urllib3 maintains a cache of idle TCP connections (the "pool"), but it's set at a fixed size. If there are no connections available from the pool, it creates a new one. When it's finished with that new connection it tries to return it to the pool, but if the pool is already full then it simply discards it. urllib3 issues a logging warning when it does so, since this essentially represents an inefficiency where the pool size is smaller than the number of simultaneous requests -- future connections will perform the full TCP connection set up. This is not a correctness issue, it's just an efficiency / performance issue.

Both urllib3 and the Python requests library that wraps it expose settings to change the pool size, but because the connections are created under the hood by boto we don't have access to these settings from the boto client (without monkeypatching or similar).

Plumbing through the pool_connections setting that the requests HTTPAdapter exposes (as mentioned in the original post on this issue) would allow us to set the pool size to whatever is appropriate for our load. It should also be easy to do, it's simply exposing a parameter and then passing that parameter value to the underlying requests library. A more general solution would be to expose overrides for all the default HTTPAdapter settings, but this is the only one I really care about.

I'll note that although this issue is probably worst for services like DynamoDB in our case we trigger the error when loading more than 10 photos from S3 simultaneously, so hopefully whatever solution you arrive at is not specific to DynamoDB.

Lastly, simply for reference if you're curious, here's the place in the urllib3 connection pool code where the warning is issued:

https://github.com/shazow/urllib3/blob/65b8c52c16dee5c3a523de2c1c21853ba0e581f2/urllib3/connectionpool.py#L257

and here's the docs for the connection pool:

https://urllib3.readthedocs.io/en/1.4/pools.html

and here's a sort of how-to article on the setting in the requests library with way more information than you probably want on the subject:

https://laike9m.com/blog/requests-secret-pool_connections-and-pool_maxsize,89/

Thanks!

[jamesls]

Thanks for the info. One last thing I noticed, while pool_connection is the number of conn pools to be cached in memory at any given point (1 conn pool per host), pool_maxsize is the total number of connections to keep in a single connection pool. pool_maxsize is what we'd need to accommodate the multithreaded scenario that's been outlined here, but I'm not sure if pool_connection matters that much (it would matter if you're accessing multiple AWS services). In requests, they use the same default value for both:

class HTTPAdapter(BaseAdapter):
    def __init__(self, pool_connections=DEFAULT_POOLSIZE,
                 pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
                 pool_block=DEFAULT_POOLBLOCK):

I wonder if we should simplify and do something similar, expose a max_poolsize and just set that value for both pool_connection and pool_maxsize.

[wiltzius]

Ah, I hadn't noticed that distinction. Indeed for our case it's the
pool_maxsize that we care about. We only ever use S3 from our production
EC2 instances .

On Wed, Sep 7, 2016 at 5:03 PM, James Saryerwinnie <[email protected]

wrote:

Thanks for the info. One last thing I noticed, while pool_connection is
the number of conn pools to be cached in memory at any given point (1 conn
pool per host), pool_maxsize is the total number of connections to keep
in a single connection pool. pool_maxsize is what we'd need to
accommodate the multithreaded scenario that's been outlined here, but I'm
not sure if pool_connection matters that much (it would matter if you're
accessing multiple AWS services). In requests, they use the same default
value for both:

class HTTPAdapter(BaseAdapter):
def init(self, pool_connections=DEFAULT_POOLSIZE,
pool_maxsize=DEFAULT_POOLSIZE, max_retries=DEFAULT_RETRIES,
pool_block=DEFAULT_POOLBLOCK):

I wonder if we should simplify and do something similar, expose a
max_poolsize and just set that value for both pool_connection and
pool_maxsize.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/boto/botocore/issues/766#issuecomment-245456600, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAO97ZZyKsmtXRPVo4JYS6Mstijp_dthks5qn1DrgaJpZM4HKQtx
.