Docker-jitsi-meet: Java processes don't use flags for correct heap sizing within docker

[bkempe]

See https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/.

The -XX:+UseCGroupMemoryLimitForHeap flag seems to be enabled by default on newer versions of openjdk:8:

docker run --rm -m 100MB openjdk:8u212 java -XshowSettings:vm -version

but the public base-java still uses an older one that would need the manual flag:

docker run --rm -m 100MB jitsi/base-java java -XshowSettings:vm -version

This can cause unexpected out of memory / crashes.

The container support has been further improved in JDK 10:

• https://bugs.openjdk.java.net/browse/JDK-8186315
• https://bugs.openjdk.java.net/browse/JDK-8189497

[saghul]

Thanks for letting us know! Do you know what version has this problem, and if it was updated in Debian stable?

[saghul]

This should be fixed with the next update.

[bkempe]

One thing to note is that the JVM max heap size currently needs to be provided as absolute Xmx value for JVB and is hard-coded in the jicofo.sh.

Dynamic heap sizing based on the container limits would be preferable, but is complicated with Java 8. Are there plans to upgrade to Java 10+?

[bkempe]

Actually, it looks like the _Java Improvements for Docker Containers_ have been backported to Java 8: https://www.oracle.com/technetwork/java/javase/8u191-relnotes-5032181.html

[saghul]

@bkempe Thanks for the details! Yes, there are plans to migrate to Java 10, but we have quite a few projects, so this may take a bit of time.

Given the support for containers was backported to Java 8 and that the version we use is 8u212, is there anything that we'd need to change still?

Thanks!

[bkempe]

Glad to hear that Java 10 (or 11 LTS?) is on the roadmap.

In order to support dynamic heap sizes depending on how much memory is available in the container, the absolute -Xmx3072m parameters (JVB and Jicofo) need to be replaced with a more flexible approach (e.g. an environment variable JVM_OPTS + a default value), so that values like -XX:MinRAMPercentage=70.0 -XX:MaxRAMPercentage=70.0 can be passed through and are not overridden by the -Xmx.

[saghul]

Hum. I suspect Jigasi would need the same. Now, does Java do The Right Thing when the passed absolute max is larger than the available memory?

If it behaves correctly, then I think we should be good for now, since there should be no need to tweak the max memory parameter, these applications are not RAM hungry.

[saghul]

ping @damencho as the in-house Java overlord.

[bkempe]

It would be very bad if the Java heap becomes larger than the physical memory because of swap memory/thrashing. Also, the memory allocated for Java should not be the maximum available physical memory.
Being able to pass memory settings without them being fixed at 3GB (e.g. Jicofo needs far less) would be very useful.

[damencho]

Jigasi is setting JIGASI_MAX_MEMORY=3072m.
And by the way, I've tested java11 and I'm currently pushing PRs so we can run using java11. Trying to push it these days, release on meet.jit.si and push it to stable.

[bkempe]

BTW, this is a helpful tool for setting the JVM memory sizes correctly within docker: https://github.com/cloudfoundry/java-buildpack-memory-calculator
https://github.com/meisterplan/jdk-base is a docker image that also uses these calculations

[everflux]

Any update on this issue? At least environment variables for the docker containers to manually set the limits would be greatly appreciated.