Pulsar: OutOfDirectMemoryError

[bputt]

Describe the bug
We're using a proxy to connect to our pulsar cluster and the proxy was failing to send messages to our broker. The error on the broker is:

WARN io.netty.channel.ChannelInitializer - Failed to initialize a channel. Closing: [id: .... ]
io.netty.util.internal.OutOfDirectMemoryError: failed to allocated 866 byte(s) of direct memory (used: 4Gb, max: 4Gb)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
at io.netty.util.internal.PlatformDependent.allocatedDirectNoCleaner(PlatformDependent.java:594)
...

Expected behavior
I'd expect the proxy to send data to another broker, but I also don't see why this broker would have an OOM when it should be persisting to disk for the most part?

Desktop (please complete the following information):
CentOS 7.6
Pulsar 2.2.1

Additional context
-Xms4g -Xmx16g -XX:MaxDirectMemorySize=4g

[merlimat]

@bputt This would be kind of addressed in 2.3. The changes were described in https://github.com/apache/pulsar/wiki/PIP-24%3A-Simplify-memory-settings .

The implementation was actually done in BookKeeper and will be available in 4.9.0 release (which is currently being voted). Once that's available, we'll add the hooks in Pulsar broker as well to ensure that under any circumstances, the broker will restart when a OOM exception occurs. (at the same mitigations for that not to happen as easily are also there).

[bputt]

@merlimat do you have an expected release date for 2.3?

[sijie]

@bputt we are blocked at bookkeeper 4.9 release. A new 4.9 rc will be sent out for review today, hopefully we can have 4.9 available early next week. so hopefully we started pulsar 2.3 release late next week.

[sautran]

@sijie, I saw this same issue when I run perf-test (I used the minikube for the pulsar setup). It is in the broker log.
Has the issue been fixed?

I followed instruction in http://pulsar.apache.org/docs/en/deploy-kubernetes/
minikube start --memory=8192 --cpus=4

[sijie]

@sautran the settings are available since 2.3.0