Akka: Large EnvelopeBuffers aren't returned to the largeEnvelopeBufferPool

[astonchev]

Our cluster runs with akka 2.4.11, artery, embedded aeron driver and several large-message-destinations. One cluster member emits large messages at a rate of 1 to 160 per second.
Depending on the amount of free RAM, -XX:+DisableExplicitGC and -XX:MaxDirectMemorySize the problem manifests itself as one of:

• [Full GC (System.gc())
• java.lang.OutOfMemoryError: Direct buffer memory
• the kernel killing the process (OOM-killer)

Investigation with a debugger and BTrace showed that within the outbound large messages flow every EnvelopeBuffer is aquired from the largeEnvelopeBufferPool but released to the regular envelopeBufferPool.
That makes largeEnvelopeBufferPool allocate a new DirectByteBuffer on every call to aquire(), and direct memory consumption rises dramatically.
Please, see suggested fix https://github.com/astonchev/akka/commit/08b974b2cf6afb5ec2524cc599573f61959afc0b to better understand what I'm talking about

[dlisin]

We have exactly the same problem in our application. I will really appreciate if fix will be included to nearest Akka release

[Evgen1000end]

Our team is also aware of this issue. We will also be very grateful if the fix will come out soon.

[patriknw]

Good catch, please open a PR if you know how to fix it.

[astonchev]

Hi @patriknw, I've opened a PR https://github.com/akka/akka/pull/22729
The fix is trivial but it took me some time to figure out how to test it. Can you please see that the test complies with your design principles.