Reactor-netty: Memory leak in PooledConnectionProvider

[eugene-sadovsky]

Expected Behavior

channelPools map should be cleaned-up of stale targets

Actual Behavior

I observe heap memory growth over a week in Spring Cloud Gateway (3.0.1) and reactor-netty-1.0.3. After analyzing the heap memory dump I found that channelPools accumulates stale targets and keeps on growing. At the time it contained close to 2k entries, most of them pointing to stale, non-empty SimpleDequePools. This happens because we use service discovery and our platform is rather dynamic, it scales up and down often. Services change their IPs/ports and this results in constantly growing channelPools map. In our case memory growth rate is around 20MB per day

Setting non-zero evictionInterval and maxIdleTime improves the situation by emptying pooled connections, but it still does not clean up channelPools.

Possible Solution

Running background connection eviction by setting evictionInterval could also evict stale hosts from channelPools, if the host points to an empty pool, or it was not accessed for longer than maxIdleTime

Your Environment

• Reactor-netty 1.0.3
• Spring Cloud Gateway 3.0.1

[violetagg]

@eugene-sadovsky We need to consider two use cases:

• The first use case is yours when you most probably do not need this empty pool anymore
• The second use case is: after the eviction the pool is empty but just after that operation, we need to create a connection for a new request. In this use case if we close the pool and remove it from channelPools then in the next moment we will need to create that pool again.

[eugene-sadovsky]

@violetagg what if we run clean-up of channelPools at a longer intervals, for example multiples of pool evictionInterval, (2 * evictionInterval or longer ). I think timing here is not as critical as in pool eviction.
At the same time we maintain lastAccessTime of each pool, probably via each acquire() invocation.
Pool drop criteria would be then: pool is empty and is inactive for longer than n*evictionInterval.

[violetagg]

@simonbasle What do you think for having such configurations on pool level?

[simonbasle]

@simonbasle What do you think for having such configurations on pool level?

I don't think there's anything to be done at pool level, other than expose the last access time on the pooledRef (which I think is already the case) => we're talking about eviction of a pool from a map maintained by reactor-netty.

[violetagg]

@simonbasle What do you think for having such configurations on pool level?

I don't think there's anything to be done at pool level, other than expose the last access time on the pooledRef (which I think is already the case) => we're talking about eviction of a pool from a map maintained by reactor-netty.

@simonbasle Yes you are right that the operation for pool removal from the map is Reactor Netty responsibility, however we need to do that based on some state of the pool. The state of the pool at the moment is not exposed i.e. Reactor Netty can acquire resources from the pool but does not have the knowledge whether the pool is empty or not, how long was the pool empty etc.

[simonbasle]

ah, yeah we can probably add some information. we would have to consider carefully which type of information, and it would have to be best effort though, because otherwise it could require too much synchronization.

[violetagg]

Marking this as blocked, because we need a change in Reactor Pool.

[simonbasle]

On pool side, tracked in reactor/reactor-pool#134 and wip in reactor/reactor-pool#135