Openj9: Use cgroup limits to adjust internal parameters

[ashu-mehra]

From wikipedia:
cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.

Cgroup provides subsystems to handle management of various system resources. Subsystems expose parameters (which can be used to set or read limits/usage) as pseudo files within the cgroup filesystem.

Docker containers use cgroups to impose constraints on resource usage by the containers. This link provides ways in which docker can control memory, CPU, or block IO a container can use.
Essentially options provided by docker is only a sub-set of the limits imposed by cgroups.

From JVM perspective, I have found three subsystems of interest and below is the list of parameters in each of those which may be of interest to JVM:

1) memory subsystem of cgroup can be used to control following parameters:
memory.limit_in_bytes # set/show limit of memory usage
memory.memsw.limit_in_bytes # set/show limit of memory+Swap usage
memory.soft_limit_in_bytes # set/show soft limit of memory usage
memory.kmem.tcp.limit_in_bytes # set/show hard limit for tcp buf memory

Documentation about memory subsystem can be seen at https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

2) cpuset subsystem of cgroup can be used to control following parameters:
cpuset.cpus: list of CPUs in that cpuset
cpuset.mems: list of Memory Nodes in that cpuset

Documentation about cpuset subsystem can be seen at https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt

3) cpu subsystem of cgroup can be used to control following parameters:
cpu.cfs_quota_us
cpu.shares

This subsystem is not well documented, but Redhat provides explanation of some of the parameters here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu

OpenJDK-Hotspot already provides support for Docker CPU and memory limits: https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
Currently they are taking into consideration "memory.limit_in_bytes" and "cpuset.cpus" for adjusting internal parameters.

There is also this JEP (in draft state) that talks about making JVM and Core libraries aware of limits imposed by containers.

OpenJ9 also uses memory and CPUs to determine internal parameters, just like OpenJDK-Hotspot does:

1. uses available physical memory returned by API omrsysinfo_get_physical_memory() (in eclipse/omr) to

   • calculate the maximum size of the object heap in absence of -Xmx option in MM_GCExtensionsBase::initialize() and MM_GCExtensions::initialize()

2. uses available CPUs returned by API omrsysinfo_get_number_CPUs_by_type() (in eclipse/omr) to

   • determine number of helper threads created by GC (MM_GCExtensionsBase::gcThreadCount)
   • determine number of compilation threads (OMR::Options::_numUsableCompilationThreads)

Ideally, when running in a cgroup (either through Docker container, or diectly on bare metal/VM), OpenJ9 should be using limits imposed on memory and CPU by the cgroup to determine above parameters.
Currently, OpenJ9 uses port library APIs (as mentioned above) in eclipse OMR to get these resource limits. Therefore, it makes sense to update the port library to support cgroup system.
This can be done in two ways:
1) Adding new port library APIs that return limits imposed by the runtime process's cgroup, and then updating the runtime to use the new APIs.
2) Updating existing API(s) to return cgroup limits. This should be protected by a flag which would be set by the runtime if it wishes to use cgroup limits. With this approach minimal changes would be required in the runtime for using the cgroup limits.

For the work required in port library, I have opened issue in eclipse OMR https://github.com/eclipse/omr/issues/1281.
I am opening this issue to discuss and track any changes required in OpenJ9 to enable and use cgroup limits via OMR port library.

Cgroup provides many more resource controllers, 12 to be precise. I have only looked at 3 of them and how they can impact JVM.
If you are aware of other resource limits that can be controlled by the cgroup and can potentially impact JVM, please feel free to comment here.

[ashu-mehra]

Came across Oracle bug https://bugs.openjdk.java.net/browse/JDK-8189497 which talks about option -XX:-UseContainerSupport to disable container awareness in JVM. It appears default behavior for OpenJDK/Hotspot is to enable container awareness.
I believe OpenJ9 should also adopt same option. But as we are in early stage of enabling container awareness, I think its better to enable container awareness only if option -XX:+UseContainerSupport is specified. Later on when the code stabilizes, we can enable it by default.

[JamesKingdon]

Does this also relate to https://github.com/eclipse/openj9/issues/1166 ?

[ashu-mehra]

Does this also relate to #1166 ?

Yes, the two are related. Issue with Runtime.availableProcessors() can be covered under this.

[PascalSchumacher]

It would be nice if this was implemented, as it seems that currently the number of compiler threads is not based on CPU limits.

Details:

We are using openjdk11-openj9:jre-11.0.4_11_openj9-0.15.1-alpine and start the JVM with -Xtune:virtualized.

We limit the number of CPUs of the container via kubernetes configuration:

resources:
  limits:
    cpu: 2

The limit seems to be working:

jjs> java.lang.Runtime.getRuntime().availableProcessors()
2

But the number of compilation threads (7) seems be bit high according to this dump generated with kill -3:

grep "JIT Compilation" javacore.20190905.081636.1.0001.txt
3LKWAITNOTIFY "JIT Compilation Thread-0" (J9VMThread:0x0000000000FDAA00)
3LKWAITNOTIFY "JIT Compilation Thread-1 Suspended" (J9VMThread:0x0000000000FDE600)
3LKWAITNOTIFY "JIT Compilation Thread-2 Suspended" (J9VMThread:0x0000000000FE2200)
3LKWAITNOTIFY "JIT Compilation Thread-3 Suspended" (J9VMThread:0x0000000000FE5E00)
3LKWAITNOTIFY "JIT Compilation Thread-4 Suspended" (J9VMThread:0x0000000000FE9A00)
3LKWAITNOTIFY "JIT Compilation Thread-5 Suspended" (J9VMThread:0x0000000000FED500)
3LKWAITNOTIFY "JIT Compilation Thread-6 Suspended" (J9VMThread:0x0000000000FF1100)
3XMTHREADINFO "JIT Compilation Thread-0" J9VMThread:0x0000000000FDAA00, omrthread_t:0x00007F01D8119890, java/lang/Thread:0x00000000F386F6E8, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-1 Suspended" J9VMThread:0x0000000000FDE600, omrthread_t:0x00007F01D8119E08, java/lang/Thread:0x00000000F386F780, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-5 Suspended" J9VMThread:0x0000000000FED500, omrthread_t:0x00007F01D811C828, java/lang/Thread:0x00000000F386F9E0, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-4 Suspended" J9VMThread:0x0000000000FE9A00, omrthread_t:0x00007F01D811C2B0, java/lang/Thread:0x00000000F386F948, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-2 Suspended" J9VMThread:0x0000000000FE2200, omrthread_t:0x00007F01D811ADA0, java/lang/Thread:0x00000000F386F818, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-3 Suspended" J9VMThread:0x0000000000FE5E00, omrthread_t:0x00007F01D811B318, java/lang/Thread:0x00000000F386F8B0, state:R, prio=10
3XMTHREADINFO "JIT Compilation Thread-6 Suspended" J9VMThread:0x0000000000FF1100, omrthread_t:0x00007F01D811D7C0, java/lang/Thread:0x00000000F386FA78, state:R, prio=10

[DanHeidinga]

@mpirvu Can you comment on the number of JIT threads here?

[ashu-mehra]

If I remember correctly we always start fixed number of Compilation Threads (I think it is 7), but not all of them are active. I think number of active compilation threads do depend on number of CPUs, among many other factors, which takes into account CPUs allocated to the container.
In the above output it can be seen only one thread JIT Compilation Thread-0 is active, rest all are suspended eg JIT Compilation Thread-1 Suspended.

@mpirvu please correct me if I am wrong.

[mpirvu]

On Linux the JIT always starts 7 compilation threads to be able to deal with starvation issues. Because a non root user cannot set raise priority of threads, an application with very many threads can starve the few existing compilation threads. Compilation threads are activated based on load (compilation queue size/weight). Typically the JIT will activate up to numProc-1 threads (1 if numProc==1), but once starvation is detected the remaining compilation threads also become eligible for activation.

[mpirvu]

If the user really wants to prevent the activation of additional compilation threads in case of starvation, he/she can use -XcompilationThreads option.

[PascalSchumacher]

Thank you very much for the detailed information! 👍