Openj9: JIT memory consumption appears to continue to be a problem with containers

[huntc]

Despite the fix referenced by https://github.com/eclipse/openj9/issues/1371, we appear to continue to experience JIT memory consumption issues that lead to being killed by the Linux OOM manager.

I understand that we are using a distribution of OpenJ9 with the newly incorporated JIT memory management functionality:

$ docker run -it ibmcom/ibmjava:8-sfj-alpine sh
/ # java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.25 - pxa6480sr5fp25-20181030_01(SR5 FP25) Small Footprint)
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181029_400846 (JIT enabled, AOT enabled)
OpenJ9   - c5c78da
OMR      - 3d5ac33
IBM      - 8c1bdc2)
JCL - 20181022_01 based on Oracle jdk8u191-b26

Here is what the OOM manager reports when it kills our app (note that the container memory allocation is set to 512MiB):

[Thu Dec 13 00:21:30 2018] JIT Compilation invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
[Thu Dec 13 00:21:30 2018] JIT Compilation cpuset=lxc-23527-fdp.libvirt-lxc mems_allowed=0
[Thu Dec 13 00:21:30 2018] CPU: 0 PID: 23660 Comm: JIT Compilation Not tainted 4.1.49-rt52-yocto-standard #1
[Thu Dec 13 00:21:30 2018] Hardware name: Lynx Software Technologies, Inc.  , BIOS Version TRUNK (ENGINEERING) 12/20/2017
[Thu Dec 13 00:21:30 2018]  0000000000000000 ffff8800377abc98 ffffffff819b8a4f ffff880037d7e100
[Thu Dec 13 00:21:30 2018]  ffff880006c6d000 ffff8800377abd48 ffffffff8112c7f5 ffff8800377abcf8
[Thu Dec 13 00:21:30 2018]  0000000000000202 ffff8800381b3f10 ffffffff81136068 00000000000000fd
[Thu Dec 13 00:21:30 2018] Call Trace:
[Thu Dec 13 00:21:30 2018]  [<ffffffff819b8a4f>] dump_stack+0x63/0x81
[Thu Dec 13 00:21:30 2018]  [<ffffffff8112c7f5>] dump_header.isra.6+0x75/0x210
[Thu Dec 13 00:21:30 2018]  [<ffffffff81136068>] ? __page_cache_release+0x28/0x130
[Thu Dec 13 00:21:30 2018]  [<ffffffff81136b1f>] ? put_page+0x3f/0x60
[Thu Dec 13 00:21:30 2018]  [<ffffffff8112cf50>] oom_kill_process+0x1c0/0x3a0
[Thu Dec 13 00:21:30 2018]  [<ffffffff81176f3f>] ? mem_cgroup_iter+0x1df/0x430
[Thu Dec 13 00:21:30 2018]  [<ffffffff811799f9>] mem_cgroup_oom_synchronize+0x579/0x5b0
[Thu Dec 13 00:21:30 2018]  [<ffffffff81176b00>] ? mem_cgroup_can_attach+0x150/0x150
[Thu Dec 13 00:21:30 2018]  [<ffffffff8112d66f>] pagefault_out_of_memory+0x1f/0xc0
[Thu Dec 13 00:21:30 2018]  [<ffffffff8104af55>] mm_fault_error+0x75/0x160
[Thu Dec 13 00:21:30 2018]  [<ffffffff8104b460>] __do_page_fault+0x420/0x430
[Thu Dec 13 00:21:30 2018]  [<ffffffff819bb519>] ? __schedule+0x2b9/0x5f0
[Thu Dec 13 00:21:30 2018]  [<ffffffff8104b492>] do_page_fault+0x22/0x30
[Thu Dec 13 00:21:30 2018]  [<ffffffff819c0fa8>] page_fault+0x28/0x30
[Thu Dec 13 00:21:30 2018] Task in /apphosting.partition/lxc-23527-fdp.libvirt-lxc killed as a result of limit of /apphosting.partition/lxc-23527-fdp.libvirt-lxc
[Thu Dec 13 00:21:30 2018] memory: usage 524288kB, limit 524288kB, failcnt 6107
[Thu Dec 13 00:21:30 2018] memory+swap: usage 524288kB, limit 9007199254740988kB, failcnt 0
[Thu Dec 13 00:21:30 2018] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu Dec 13 00:21:30 2018] Memory cgroup stats for /apphosting.partition/lxc-23527-fdp.libvirt-lxc: cache:856KB rss:523432KB rss_huge:0KB mapped_file:128KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:523428KB inactive_file:440KB active_file:416KB unevictable:4KB
[Thu Dec 13 00:21:30 2018] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[Thu Dec 13 00:21:30 2018] [23527]     0 23527    19793     1756      41       3        0             0 libvirt_lxc
[Thu Dec 13 00:21:30 2018] [23529] 100000 23529      395       31       6       3        0             0 startcontainer.
[Thu Dec 13 00:21:30 2018] [23562] 100000 23562   481477   129889     353       6        0             0 java
[Thu Dec 13 00:21:30 2018] [23658] 100000 23658    41674      945      49       3        0             0 virsh
[Thu Dec 13 00:21:30 2018] [23659] 100000 23659      393        1       5       3        0             0 sh
[Thu Dec 13 00:21:30 2018] Memory cgroup out of memory: Kill process 23562 (java) score 993 or sacrifice child
[Thu Dec 13 00:21:30 2018] Killed process 23562 (java) total-vm:1925908kB, anon-rss:519556kB, file-rss:0kB

...and here are the relevant JVM options:

-XX:+UseContainerSupport
-XX:MaxRAMPercentage=30
-Xss384k
-Xscmx24m

This issue is a bit of a show-stopper for us, so I really appreciate the attention it may receive.

[DanHeidinga]

FYI @mpirvu

[mpirvu]

I will look into it. @huntc could you please give me some details about your OS?
Also, do you allow the container to use swap?

[huntc]

This is running on Cisco's IOx, which is a Yocto build. I don't believe swap is being used. From inside the container:

/ # cat /proc/meminfo
MemTotal:         959872 kB
MemFree:           15888 kB
MemAvailable:     524628 kB
Buffers:           45464 kB
Cached:           445496 kB
SwapCached:            0 kB
Active:           653216 kB
Inactive:         216376 kB
Active(anon):     384336 kB
Inactive(anon):      396 kB
Active(file):     268880 kB
Inactive(file):   215980 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                24 kB
Writeback:             0 kB
AnonPages:        378668 kB
Mapped:            46124 kB
Shmem:              6100 kB
Slab:              51964 kB
SReclaimable:      38532 kB
SUnreclaim:        13432 kB
KernelStack:        3088 kB
PageTables:         4892 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      479936 kB
Committed_AS:     921272 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        2560 kB
VmallocChunk:   34359725976 kB
DirectMap4k:       19456 kB
DirectMap2M:      974848 kB

[mpirvu]

I am not familiar with Yocto; I see it targets the embedded landscape.
Nevertheless, could you please add -Xjit:verbose={compilePerformance},vlog=pathToYourVlogFile to the Java command line options, run your test and send me the verbose file that gets generated?
To avoid generating a new image you could pass options through IBM_JAVA_OPTIONS environment variable. For instance:
docker run -e IBM_JAVA_OPTIONS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=30 -Xss384k -Xscmx24m -Xjit:verbose={compilePerformance},vlog=/tmp/vlog.txt" yourImage

[huntc]

FYI I've been trying to create a reproducer via Docker for you. So far, I can't reproduce the issue. I'm going to try again with our target environment, which uses LXC. Indeed, could it be that LXC does things a little differently to Docker w.r.t. cgroups and memory limits?

[mpirvu]

@ashu-mehra Ashutosh, does the container support you implemented in omr/OpenJ9 include LXC or is it limited to Docker?

[ashu-mehra]

We have not tried it on LXC yet. It was implemented considering Docker only. I won't be surprised if it doesn't work with LXC. In fact there are few other container system out there for which current mechanism may not work. I think we should take them up on case-by-case basis.
If LXC support is required, we should definitely look into adding it.

[huntc]

LXC is definitely required by us. :-) I’ll try and get you some more data. Thanks.

[mpirvu]

@ashu-mehra Ashutosh, are you planning to add LXC support in the near future? If so, could you please provide a rough estimate of the timeline? Thanks

[deesebas]

Hi @huntc can you provide the data that you had mentioned regarding your need to use OpenJ9 within LXC instead of Docker containers?

[huntc]

Yes, I shall try this again. Sorry that I’ve been a bit busy. I owe you the data.

[ashu-mehra]

I identified couple of issues from code walkthrough when OpenJ9 is running in lxc container, but I am not sure they can cause JIT to use memory beyond container limit.

1) isRunningInContainer is specific to Docker containers and the mechanism used won't work for identifying an lxc container.
This documentation page https://lxd.readthedocs.io/en/latest/container-environment/ says PID 1 gets an environment variable "container=lxc". I think the same would be available to all other processes as well, unless the environment is specifically modified by the parent process.
So a check for container=lxc environment variable can be added in the function isRunningInContainer to detect lxc container.

2) Secondly the API omrsysinfo_get_memory_info is expected to return memory deatils of the host as well as the container. It reads host memory details from /proc/meminfo and container stats from cgroups. This works fine for docker container where /proc is same as in the host. But fails for lxc container, because lxc (or lxd?) mounts LXCFS on /proc/meminfo which provides CGroup-aware stats. So we can't fetch memory stats of the host when running in lxc container.
However, this should not prevent JIT from getting correct values of available memory in the container.

I will add few debug statements to verify omrsysinfo_get_memory_info is returning expected values.

@huntc do you have any sample app that reproduces the issue when running in lxc container?

[huntc]

I have something of a reproducer, although it isn't exactly the same. I'm using the Spring Pet Clinic app as it is widely available, unlike our app which I cannot share.

Here are the steps that I take:

1. Disabled swap on my Linux host as my lxc setup on a VM runs privileged containers (unprivileged containers will require an additional memsw limit as per step 5)
2. Created an lxc container using Ubuntu (sudo lxc-create -n playtime -t download -- --dist ubuntu --release xenial --arch amd64)
3. Attached to the container and:
   2.1 Installed the OpenJ9 JDK
   2.2 Built the Spring pet-clinic example (build steps)
   2.3 Installed time using apt install time
4. Ran pet-clinic within the container using /usr/bin/time/time and noted the peak RSS was 244804 KiB given no additional options to the java command
5. Stopped the container from another terminal
6. Edited the containers /var/lib/lxc/<container-name>/config and declared lxc.cgroup.memory.limit_in_bytes = 134217728 (128MiB - deliberately way less than the peak RSS, but hopefully still a reasonable size for the app even though it may suffer memory pressure)
7. Ran the app within the container again: java -XX:+UseContainerSupport -jar target/*.jar

The app is then killed by the OOM.

Again, this isn't exactly the same as what I observed on my small machine (which isn't available to me right now). However, my expectation is that the Linux OOM manager should rarely kick in given the JVM's UseContainerSupport param and, instead, I should expect the JVM's OutOfMemory error to surface.

As additional info, here's the OOM output:

[Thu Jan 31 15:18:45 2019] main invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=0
[Thu Jan 31 15:18:45 2019] main cpuset=playtime mems_allowed=0
[Thu Jan 31 15:18:45 2019] CPU: 1 PID: 9033 Comm: main Not tainted 4.4.0-87-generic #110-Ubuntu
[Thu Jan 31 15:18:45 2019] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[Thu Jan 31 15:18:45 2019]  0000000000000286 8c51ec59d0628c41 ffff88002976fc88 ffffffff813f9903
[Thu Jan 31 15:18:45 2019]  ffff88002976fd68 ffff880077551c00 ffff88002976fcf8 ffffffff8120b75e
[Thu Jan 31 15:18:45 2019]  ffffffff81cd9828 ffff880078293c00 ffffffff81e67720 0000000000000206
[Thu Jan 31 15:18:45 2019] Call Trace:
[Thu Jan 31 15:18:45 2019]  [<ffffffff813f9903>] dump_stack+0x63/0x90
[Thu Jan 31 15:18:45 2019]  [<ffffffff8120b75e>] dump_header+0x5a/0x1c5
[Thu Jan 31 15:18:45 2019]  [<ffffffff81192ce2>] oom_kill_process+0x202/0x3c0
[Thu Jan 31 15:18:45 2019]  [<ffffffff811ff384>] ? mem_cgroup_iter+0x204/0x390
[Thu Jan 31 15:18:45 2019]  [<ffffffff812013e3>] mem_cgroup_out_of_memory+0x2b3/0x300
[Thu Jan 31 15:18:45 2019]  [<ffffffff812021b8>] mem_cgroup_oom_synchronize+0x338/0x350
[Thu Jan 31 15:18:45 2019]  [<ffffffff811fd290>] ? kzalloc_node.constprop.49+0x20/0x20
[Thu Jan 31 15:18:45 2019]  [<ffffffff81193394>] pagefault_out_of_memory+0x44/0xc0
[Thu Jan 31 15:18:45 2019]  [<ffffffff8106b302>] mm_fault_error+0x82/0x160
[Thu Jan 31 15:18:45 2019]  [<ffffffff8106b7b8>] __do_page_fault+0x3d8/0x400
[Thu Jan 31 15:18:45 2019]  [<ffffffff8106b802>] do_page_fault+0x22/0x30
[Thu Jan 31 15:18:45 2019]  [<ffffffff81844038>] page_fault+0x28/0x30
[Thu Jan 31 15:18:45 2019] Task in /lxc/playtime killed as a result of limit of /lxc/playtime
[Thu Jan 31 15:18:45 2019] memory: usage 131072kB, limit 131072kB, failcnt 39
[Thu Jan 31 15:18:45 2019] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu Jan 31 15:18:45 2019] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu Jan 31 15:18:45 2019] Memory cgroup stats for /lxc/playtime: cache:8268KB rss:122804KB rss_huge:53248KB mapped_file:468KB dirty:0KB writeback:0KB inactive_anon:8236KB active_anon:122768KB inactive_file:0KB active_file:0KB unevictable:4KB
[Thu Jan 31 15:18:45 2019] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[Thu Jan 31 15:18:45 2019] [ 8757]     0  8757     9294     1326      23       3        0             0 systemd
[Thu Jan 31 15:18:45 2019] [ 8854]     0  8854     8817      747      19       3        0             0 systemd-journal
[Thu Jan 31 15:18:45 2019] [ 8903]     0  8903     7244      754      20       3        0             0 cron
[Thu Jan 31 15:18:45 2019] [ 8909]   104  8909    64097      804      27       3        0             0 rsyslogd
[Thu Jan 31 15:18:45 2019] [ 8983]     0  8983     4029      214      12       3        0             0 dhclient
[Thu Jan 31 15:18:45 2019] [ 9007]     0  9007     3937      501      14       3        0             0 agetty
[Thu Jan 31 15:18:45 2019] [ 9008]     0  9008     3937      553      13       3        0             0 agetty
[Thu Jan 31 15:18:45 2019] [ 9009]     0  9009     3937      550      13       3        0             0 agetty
[Thu Jan 31 15:18:45 2019] [ 9010]     0  9010     3937      543      14       3        0             0 agetty
[Thu Jan 31 15:18:45 2019] [ 9011]     0  9011     3937      545      13       3        0             0 agetty
[Thu Jan 31 15:18:45 2019] [ 9028]     0  9028     5294      962      16       3        0             0 bash
[Thu Jan 31 15:18:45 2019] [ 9032]     0  9032   428530    34814     135       5        0             0 java
[Thu Jan 31 15:18:45 2019] Memory cgroup out of memory: Kill process 9032 (java) score 1034 or sacrifice child

HTH.

[ashu-mehra]

@huntc I am not able to reproduce the issue with Sprint Pet Clinic app. Followed same steps as you described in your last comment.

I used OpenJ9 binary from Adopt: https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u202-b08/OpenJDK8U-jdk_x64_linux_openj9_linuxXL_8u202b08-openj9-0.12.0.tar.gz
java version is:

root@playtime:~/spring-petclinic# $JAVA_HOME/bin/java -version
openjdk version "1.8.0_202"
OpenJDK Runtime Environment (build 1.8.0_202-b08)
Eclipse OpenJ9 VM (build openj9-0.12.0, JRE 1.8.0 Linux amd64-64-Bit 20190130_194 (JIT enabled, AOT enabled)
OpenJ9   - 04890c300
OMR      - d2f4534b
JCL      - 6501440d7a based on jdk8u202-b08)

Initially I ran the application inside the container which has no memory limit set:

# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
9223372036854771712

Running the app using /usr/bin/time without any external java options, I get following stats after program has exited:

27.71user 2.05system 0:26.27elapsed 113%CPU (0avgtext+0avgdata 181800maxresident)k
0inputs+8outputs (0major+51379minor)pagefaults 0swaps

Max RSS is 177 MB.

I then updated container's memory limit to 128 MB (=134217728 bytes) using following command:

lxc-cgroup -n playtime memory.limit_in_bytes "134217728"

Inside the container:

# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
134217728

With this setting I ran the application again multiple times with and without -XX:+UseContainerSupport (actually this option is enabled by default, so it shouldn't really matter), but the container never got killed.
/usr/bin/time reported this at the end when I terminated the application:

29.73user 4.81system 0:29.01elapsed 119%CPU (0avgtext+0avgdata 122004maxresident)k
481240inputs+8outputs (8815major+90655minor)pagefaults 0swaps

Max RSS is 119 MB

Can you tell me the OpenJ9 version you used?

[huntc]

Hi @ashu-mehra. That’s a large number of available memory bytes. Do you have swapping enabled? If so then either disable it or set the memsw setting as well.

Otherwise, lessen the limit further.

Please let me know how you go.

[deesebas]

@huntc thanks for the recreate steps and data. While we continue working on this, it would be really useful for us to understand why your team prefers LXC over Docker containers for running java applications. Are there any specific scenarios where LXC is the best or only choice?

[huntc]

Howdy! We are stuck with LXC as our target is Cisco’s hardware. If you’re deploying software to anything of Cisco then LXC is the only choice.

[deesebas]

Good to know this, thanks @huntc

[ashu-mehra]

Do you have swapping enabled?

@huntc you are right. For some reason, memory controller didn't have swap related files, but still the swap was on. Once I disabled it on my ubuntu VM, I can see the java process getting killed when I reduce the memory to 128MB.
I am debugging to see what is going wrong.
Once again, thanks for those steps.

[ashu-mehra]

I added few debug statements in the code path where JIT is checking for available memory in the container, and it all seems to be working fine.
Portlibrary API omrsysinfo_get_memory_info() is returning memory stats of the container and the JIT is correctly calculating the available memory.

I ran the application again with 256MB of container memory to figure out native memory requirement of the application (and JVM).
From verbose gc logs (enabled with -Xverbosegclog:gc.log) I noticed, after the application has started up, committed java heap is 75235328 bytes (~ 72 MB), out of which 26455024 bytes is free. That means java heap is contributing around 47 MB to the RES size of the process. Total RES as reported by top at that time is 162 MB.
So the JVM's internal memory requirements (like memory used by JIT, memory for threads, classes etc) is around 115 MB.
If we run the container with only 128 MB memory, then JVM is sure to run out of native memory at which point it is likely to get killed by kernel OOM killer.

I see similar behavior with OpenJDK-Hotspot build as well. To track native memory usage I started the application with -XX:NativeMemoryTracking=summary option. For hotspot I had to increase the container memory to 300 MB. After application started up, I used jcmd <PID> VM.native_memory to see the native memory breakup.
It showed committed memory as 221773 KB, out of which committed java heap was 74456 KB. Rest of the components contributed to around 147317 (~144 MB).

So from these experiments it looks like memory requirement of the JVM (internal memory usage + java heap usage) for this particular application cannot be satisfied with 128 MB of container memory limit.

[huntc]

Thanks for the investigations. As stated, I hadn’t managed to reproduce the exact issue I had before, and wondered if the one I provided here would be sufficient. It isn’t though and what you write makes sense to me.

I’ve got the target device available to me again so I can try to get more info for you. Thanks for hanging in there.

[ashu-mehra]

From verbose gc logs (enabled with -Xverbosegclog:gc.log) I noticed, after the application has started up, committed java heap is 75235328 bytes (~ 72 MB), out of which 26455024 bytes is free. That means java heap is contributing around 47 MB to the RES size of the process. Total RES as reported by top at that time is 162 MB.
So the JVM's internal memory requirements (like memory used by JIT, memory for threads, classes etc) is around 115 MB.

I stand corrected here. Whole of committed java heap of 72 MB contributes to the RES size of the process, not just the used heap. So the JVM's internal memory requirements are around 90 MB.
Having said this, final conclusion remains the same, only the figures changed a bit.

[pverkest]

Hi, I seem to have a similar problem.

I'm using OpenJ9 from https://hub.docker.com/r/adoptopenjdk/openjdk8-openj9 version jdk8u202-b08.

My java version:

openjdk version "1.8.0_202"
OpenJDK Runtime Environment (build 1.8.0_202-b08)
Eclipse OpenJ9 VM (build openj9-0.12.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190130_202 (JIT enabled, AOT enabled)
OpenJ9   - 04890c300
OMR      - d2f4534b
JCL      - 6501440d7a based on jdk8u202-b08)

I'm running on Kubernetes and give my container 384 MiB of memory and these are the relevant options I'm passing as arguments: -Xjit:verbose={compilePerformance} -Xaot:verbose={compilePerformance} -Xtune:virtualized -verbose:init -verbose:sizes -verbose:gc -Xmx128m -Xcodecachetotal16m -Xquickstart -Xaot

Typical memory usage of one of my containers looks like:

0SECTION       NATIVEMEMINFO subcomponent dump routine
NULL           =================================
0MEMUSER
1MEMUSER       JRE: 505,934,024 bytes / 16041 allocations
1MEMUSER       |
2MEMUSER       +--VM: 439,694,416 bytes / 14080 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Classes: 90,176,592 bytes / 5401 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Memory Manager (GC): 140,571,016 bytes / 1369 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Java Heap: 134,279,168 bytes / 1 allocation
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 6,291,848 bytes / 1368 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Threads: 32,284,112 bytes / 496 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Java Stack: 1,791,648 bytes / 89 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Native Stack: 29,884,416 bytes / 90 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 608,048 bytes / 317 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Trace: 807,848 bytes / 485 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JVMTI: 17,776 bytes / 13 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JNI: 2,101,672 bytes / 5381 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Port Library: 172,288,840 bytes / 131 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Unused <32bit allocation regions: 172,269,104 bytes / 1 allocation
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 19,736 bytes / 130 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Other: 1,446,560 bytes / 804 allocations
1MEMUSER       |
2MEMUSER       +--JIT: 30,197,920 bytes / 668 allocations
2MEMUSER       |  |
3MEMUSER       |  +--JIT Code Cache: 16,777,216 bytes / 1 allocation
2MEMUSER       |  |
3MEMUSER       |  +--JIT Data Cache: 4,194,432 bytes / 2 allocations
2MEMUSER       |  |
3MEMUSER       |  +--Other: 9,226,272 bytes / 665 allocations
1MEMUSER       |
2MEMUSER       +--Class Libraries: 35,835,240 bytes / 281 allocations
2MEMUSER       |  |
3MEMUSER       |  +--VM Class Libraries: 35,835,240 bytes / 281 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--sun.misc.Unsafe: 34,814,656 bytes / 33 allocations
4MEMUSER       |  |  |  |
5MEMUSER       |  |  |  +--Direct Byte Buffers: 83,424 bytes / 18 allocations
4MEMUSER       |  |  |  |
5MEMUSER       |  |  |  +--Other: 34,731,232 bytes / 15 allocations
3MEMUSER       |  |  |
4MEMUSER       |  |  +--Other: 1,020,584 bytes / 248 allocations
1MEMUSER       |
2MEMUSER       +--Unknown: 206,448 bytes / 1012 allocations

If I understand it correcly, I don't have to take the 172 MB 'Port Library' into account, so I'd be using 335 MB out of the 384 MiB available in this instance.

With this configuration, some containers continue to run successfully, while others are OOM-killed once in a while by the Kubelet:

Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225918] JIT Compilation invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=952
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225921] JIT Compilation cpuset=19ff0ec7aacf066ea447b5f378a2ec7668bbe57a2b3a0ab4e1c5d36220a606fa mems_allowed=0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225926] CPU: 3 PID: 120669 Comm: JIT Compilation Not tainted 4.4.0-119-generic #143-Ubuntu
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225928] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225930]  0000000000000286 8fe875ef5fc32979 ffff880111287c88 ffffffff81400443
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225932]  ffff880111287d68 ffff880146810000 ffff880111287cf8 ffffffff8121086e
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225934]  ffff8802356d72c0 ffff880111287cc8 ffffffff8119696b ffff880005638e00
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225936] Call Trace:
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225968]  [<ffffffff81400443>] dump_stack+0x63/0x90
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225982]  [<ffffffff8121086e>] dump_header+0x5a/0x1c5
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225990]  [<ffffffff8119696b>] ? find_lock_task_mm+0x3b/0x80
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225992]  [<ffffffff81196f32>] oom_kill_process+0x202/0x3c0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225996]  [<ffffffff81204364>] ? mem_cgroup_iter+0x204/0x3a0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.225999]  [<ffffffff812063d3>] mem_cgroup_out_of_memory+0x2b3/0x300
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226001]  [<ffffffff812071bd>] mem_cgroup_oom_synchronize+0x33d/0x350
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226003]  [<ffffffff81202210>] ? kzalloc_node.constprop.49+0x20/0x20
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226005]  [<ffffffff811975e4>] pagefault_out_of_memory+0x44/0xc0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226014]  [<ffffffff8106c4e2>] mm_fault_error+0x82/0x160
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226016]  [<ffffffff8106c998>] __do_page_fault+0x3d8/0x400
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226018]  [<ffffffff8106c9e2>] do_page_fault+0x22/0x30
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226030]  [<ffffffff81852118>] page_fault+0x28/0x30
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226044] Task in /kubepods/burstable/pod9ae518f9-34df-11e9-bf1a-005056b81abc/19ff0ec7aacf066ea447b5f378a2ec7668bbe57a2b3a0ab4e1c5d36220a606fa killed as a result of limit of /kubepods/burstable/pod9ae518f9-34df-11e9-bf1a-005056b81abc
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226049] memory: usage 393216kB, limit 393216kB, failcnt 78
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226050] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226051] kmem: usage 8100kB, limit 9007199254740988kB, failcnt 0
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226052] Memory cgroup stats for /kubepods/burstable/pod9ae518f9-34df-11e9-bf1a-005056b81abc: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226076] Memory cgroup stats for /kubepods/burstable/pod9ae518f9-34df-11e9-bf1a-005056b81abc/1d562f4c47f509c676405409c1ce61ea625e75e3342f77e82ea21895702988d3: cache:4KB rss:40KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:4KB active_anon:40KB inactive_file:0KB active_file:0KB unevictable:0KB
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226098] Memory cgroup stats for /kubepods/burstable/pod9ae518f9-34df-11e9-bf1a-005056b81abc/19ff0ec7aacf066ea447b5f378a2ec7668bbe57a2b3a0ab4e1c5d36220a606fa: cache:44KB rss:385028KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB inactive_anon:20KB active_anon:385012KB inactive_file:0KB active_file:0KB unevictable:4KB
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226127] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226282] [120498]     0 120498      256        1       4       2        0          -998 pause
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226286] [120627]     0 120627     1160      212       8       3        0           952 sh
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226287] [120661]     0 120661   670965   102110     310       7        0           952 java
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.226290] Memory cgroup out of memory: Kill process 120661 (java) score 1990 or sacrifice child
Feb 20 08:18:28 BE-PLW-K8S-0004 kernel: [580439.227853] Killed process 120661 (java) total-vm:2683860kB, anon-rss:383100kB, file-rss:25340kB

Whenever this happens, my logs seem to end with:

 (cold) Compiling org/springframework/boot/actuate/health/AbstractHealthAggregator.aggregate(Ljava/util/Map;)Lorg/springframework/boot/actuate/health/Health;  OrdinaryMethod j9m=0000000002DC6698 t=664868 compThread=0 memLimit=1572864 KB freePhysicalMemory=144 MB
+ (cold) org/springframework/boot/actuate/health/AbstractHealthAggregator.aggregate(Ljava/util/Map;)Lorg/springframework/boot/actuate/health/Health; @ 00007FBF36649E40-00007FBF3664A4E4 OrdinaryMethod - Q_SZ=1 Q_SZI=1 QW=4 j9m=0000000002DC6698 bcsz=60 time=2710us mem=[region=1728 system=16384]KB compThread=0 CpuLoad=54%(13%avg) JvmCpu=0%
 (cold) Compiling org/apache/tomcat/util/http/MimeHeaders.clear()V  OrdinaryMethod j9m=0000000003F6B998 t=664868 compThread=0 memLimit=262144 KB freePhysicalMemory=144 MB
+ (cold) org/apache/tomcat/util/http/MimeHeaders.clear()V @ 00007FBF3664A540-00007FBF3664A647 OrdinaryMethod - Q_SZ=0 Q_SZI=0 QW=2 j9m=0000000003F6B998 bcsz=31 time=621us mem=[region=1024 system=16384]KB compThread=0 CpuLoad=54%(13%avg) JvmCpu=0%
 (cold) Compiling java/util/concurrent/CompletableFuture$UniCompletion.<init>(Ljava/util/concurrent/Executor;Ljava/util/concurrent/CompletableFuture;Ljava/util/concurrent/CompletableFuture;)V  OrdinaryMethod j9m=0000000003F59748 t=674877 compThread=0 memLimit=262144 KB freePhysicalMemory=144 MB
+ (cold) java/util/concurrent/CompletableFuture$UniCompletion.<init>(Ljava/util/concurrent/Executor;Ljava/util/concurrent/CompletableFuture;Ljava/util/concurrent/CompletableFuture;)V @ 00007FBF3664A6A8-00007FBF3664A74C OrdinaryMethod - Q_SZ=2 Q_SZI=2 QW=6 j9m=0000000003F59748 bcsz=20 time=1619us mem=[region=1088 system=16384]KB compThread=0 CpuLoad=147%(36%avg) JvmCpu=0%
 (cold) Compiling java/util/concurrent/CompletableFuture.postFire(Ljava/util/concurrent/CompletableFuture;I)Ljava/util/concurrent/CompletableFuture;  OrdinaryMethod j9m=00000000037E8AB8 t=674877 compThread=0 memLimit=262144 KB freePhysicalMemory=144 MB
+ (cold) java/util/concurrent/CompletableFuture.postFire(Ljava/util/concurrent/CompletableFuture;I)Ljava/util/concurrent/CompletableFuture; @ 00007FBF3664A7A8-00007FBF3664A87B OrdinaryMethod - Q_SZ=2 Q_SZI=2 QW=6 j9m=00000000037E8AB8 bcsz=59 time=1700us mem=[region=1088system=16384]KB compThread=0 CpuLoad=147%(36%avg) JvmCpu=0%
 (cold) Compiling java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V  OrdinaryMethod j9m=0000000003A0A1E0 t=674877 compThread=0 memLimit=262144 KB freePhysicalMemory=144 MB
Killed

Most of them are killed while trying to JIT compile the ForkJoinPool.managedBlock (but there are other methods taking a lot of memory too).

When searching for the JIT log line in the containers that keep on running correctly, I see this:

 (cold) Compiling java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V  OrdinaryMethod j9m=000000000203E360 t=673511 compThread=0 memLimit=262144 KB freePhysicalMemory=146 MB
! java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V time=4932659us compilationHeapLimitExceeded memLimit=262144 KB freePhysicalMemory=3 MB mem=[region=180224 system=180224]KB
 (no-opt) Compiling java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V  OrdinaryMethod j9m=000000000203E360 t=678515 compThread=0 memLimit=262144 KB freePhysicalMemory=3 MB
! java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V time=172913us compilationHeapLimitExceeded memLimit=262144 KB freePhysicalMemory=3 MB mem=[region=16384 system=16384]KB

Looking at the above log, I seem to be really on the edge of being killed here.

Is my interpretation correct that somehow the JIT thinks it can use up to 262144 KB (memLimit) and that sometimes it uses 1880224KB indeed (mem=[region=180224 system=180224]KB)? Shouldn't this have been fixed by #1371 when taking into account that I only have 384MiB available and a heap of 128MiB?

Is there anything I can change to limit the memory consumed by the JIT (already tried to lower the level to noOpt, excluding methods, ....)?

[pverkest]

Or is the JIT compiler allowed to use the Port Library and is that one too big in my case?

[ashu-mehra]

NATIVEMEMINFO section in javacore only shows virtual memory break up which does not necessarily reflect physical memory usage.

(no-opt) Compiling java/util/concurrent/ForkJoinPool.managedBlock(Ljava/util/concurrent/ForkJoinPool$ManagedBlocker;)V OrdinaryMethod j9m=000000000203E360 t=678515 compThread=0 memLimit=262144 KB freePhysicalMemory=3 MB

It does appear from the log that you are running a bit low on physical memory. It is possible that memory requirement of the JVM for running your application are more than 384MB that you have assigned to the container. Have you tried running with higher container limit, like 512MB?

@mpirvu do you think limiting scratch space used by JIT during compilation can probably be helpful in avoiding the container getting killed in such containers with low memory?

[mpirvu]

@pverkest

If I understand it correcly, I don't have to take the 172 MB 'Port Library' into account, so I'd be using 335 MB out of the 384 MiB available in this instance.

'Port library' memory should be taken into account. Could you please let me know what made you think otherwise? If there is some misleading information somewhere we need to correct it.

In the first vlog snippet the JVM sees a reasonable amount of free physical memory (freePhysicalMemory=144 MB), but those ForkJoinPool compilations appear to take a lot of memory. I think there is a bug in the optimizer because recently I saw such failures even when I allowed them to use 2GB of memory (I had 10 GB of free RAM on my machine at the time). I'll look into it.

The second vlog snippet indicates that at times your container runs very low on physical memory: freePhysicalMemory=3 MB

[pverkest]

It does appear from the log that you are running a bit low on physical memory. It is possible that memory requirement of the JVM for running your application are more than 384MB that you have assigned to the container. Have you tried running with higher container limit, like 512MB?

Yes indeed, I'm running low on physical memory, but that's the goal, since we want to have 100 of those instances. Running them with 512MB is what we now intend to do, but with a heap of (only) 128 MB, I'd hoped that the 'overhead' wouldn't be 3 times as big. Another possibility would be to disable JIT/AOT.

[huntc]

I think the goal should be to witness java.lang.OutOfMemoryError when the JVM heap is exhausted... not have the Linux OOM kill the process.

[pverkest]

'Port library' memory should be taken into account. Could you please let me know what made you think otherwise? If there is some misleading information somewhere we need to correct it.

The 'unused' together with the fact that I only gave it 384MB. I didn't know the NATIVEMEMINFO shows virtual memory.

In the first vlog snippet the JVM sees a reasonable amount of free physical memory (freePhysicalMemory=144 MB), but those ForkJoinPool compilations appear to take a lot of memory.

Despite having 144MB of free physical memory here, the Kubelet OOM killed the container because the JIT used more than those 144 MB.

Is there any way of limiting the amount of memory taken by the JIT/AOT? I'd like to have it as deterministic as possible, so that I know my container won't be killed by OOM without giving it loads of extra memory.

[pverkest]

I think the goal should be to witness java.lang.OutOfMemoryError when the JVM heap is exhausted... not have the Linux OOM kill the process.

Well, in my case it's not the JVM heap that's exhausted, so I don't expect a java.lang.OutOfMemoryError (and it's not the Linux OOM killing the process, but the kubelet).

[pverkest]

It seems to be fixed in openj9-0.12.1, but I'd still like to know whether I can limit the amount of memory the JIT/AOT takes in any way?

[ashu-mehra]

I'd still like to know whether I can limit the amount of memory the JIT/AOT takes in any way?

@mpirvu is -Xjit:scratchSpaceLimit=<in KB> something that can be used for this?

[mpirvu]

Yes, -Xjit:scratchSpaceLimit= can be used to cap the amount of scratch memory used by the JIT compiler. There is another mechanism that should kick in though: when the JIT needs to allocate a new 16MB segment for its needs, it check the available amount of physical memory and aborts the compilation in progress if satisfying that request would reduce the amount of free physical memory too much (for 384MB that safe value is 4MB)

[mpirvu]

@pverkest You indicated that your command line is -Xjit:verbose={compilePerformance} -Xaot:verbose={compilePerformance} -Xtune:virtualized -verbose:init -verbose:sizes -verbose:gc -Xmx128m -Xcodecachetotal16m -Xquickstart -Xaot and I would like to offer a few suggestions on this topic:
1) You don't need to specify verbose={compilePerformance} twice (once under -Xjit and once under -Xaot). Just once under -Xjit is enough
2) You specified both -Xtune:virtualized and -Xquickstart. The latter is a more conservative option (from JIT optimization point of view) and it will take effect over -Xtune:virtualized. Maybe that's exactly what you want given the focus on memory consumption.
3) You specified -Xaot, but, currently, in order for AOT code to be generated one must specify -Xshareclasses option as well. The shared class cache (SCC) is the repository where the AOT code is stored, so AOT cannot exist without a SCC. More information can be found here: https://www.eclipse.org/openj9/docs/xshareclasses. Also, please note that the first run with SCC will be slower because the cache is empty and needs to be populated (this is the 'cold' run). If you place the SCC inside a container such that the SCC disappears when the container goes away, then every run is going to look like a cold run. To ensure SCC persistence after your container dies one must use a volume container or map the SCC directory to a directory outside the container. For some examples please look at usage suggestions for liberty app server in containers here https://hub.docker.com/_/websphere-liberty under the Using IBM JRE Class data sharing heading.

[huntc]

Is there anything in openj9-0.12.1 or prior that could have improved container memory dimensioning?

On 25 Feb 2019, at 17:49, pverkest [e-mail] wrote:

openj9-0.12.1

[pawelebe]

hello

is there any update on that matter?

We have dockerized java application, it was working fine with 384mb container on java 8 hotspot. Recently we moved to java 11 j9. We can see that average memory consumption is much lower then before. However once on a while container got killed by oom. There is high memory usage spike due to JIT.

App is working on docker & k8s hosted on AWS.

I saw you recommended setting -Xjit:scratchSpaceLimit=, but can JVM just scale it? If not, what may be reasonable size for this setting?

[mpirvu]

@pawelebe
The default value of the scratchSpace (memory used temporarily for JIT compilations) is 256MB. -Xjit:scratchSpaceLimit= can be used to change the default value. Reasonable sizes could be 8M-32M especially if you use -Xquickstart like you indicated above. Note that the value for the option above is expressed in KB, so if you want a 16M limit you would say -Xjit:scratchSpaceLimit=16384

Scratch space limit does not scale with the memory limit set for the container. However, there is an option that could adjust the scratch space based on the readout of the available physical memory: -Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile. If you end up using this option please share with us the outcome.
Also, do you have swap space enabled or disabled?

[pawelebe]

@mpirvu many thanks for your very helpful response.

No we do not have swap space enabled.

I have tried with option you recommended Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile. So far so good, no oom since then.

Is it safe to use it as default too all our micro-services? (not matter what is RAM quota) . Are there any performance or other consequences when using this flag? Should we check anything else?

I am happy to provide you any other details if needed.

[mpirvu]

Is it safe to use it as default too all our micro-services?

It is safe to use regardless of the RAM quota. The downside is that some compilations may fail if the logic detects there isn't enough free physical memory. In contrast, without that option compilations would be attempted relying on swap space. The verbose log will show whether compilations are failing.

If you can distill your application into something simple that you could send us I'd be happy to try to reproduce the issue.

[Herr-Sepp]

Hello, I am also afraid that my rather aggressive memory limit of the docker container will be exceeded by the JIT compilations.

I cannot find these options in the documentation
Xjit:scratchSpaceLimit=
Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile
has it been replaced by -Xcodecachetotal?

Reasonable sizes could be 8M-32M especially if you use -Xquickstart like you indicated above

Why is it possible to reduce the cache so much from 256M to 32M with -Xquickstart?

And why is Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile not set automatically with the option -Xtune:virtualized? Especially for virtualized environments it is very important that the free memory is not exceeded, because in K8s and openshift there is no swap. Exceeding the limit always resulted in an OOM kill.

Thank you

[mpirvu]

@Herr-Sepp

I cannot find these options in the documentation
Xjit:scratchSpaceLimit=
Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile

Options under -Xjit: heading are not supported and therefore they are not documented. They are mostly used for diagnostic or as workarounds for problems in the field.

While those two options refer to the transient memory used by the JIT during compilation, -Xcodecachetotal serves a totally different purpose and refers to the repository where jitted/compiled code resides.

Why is it possible to reduce the cache so much from 256M to 32M with -Xquickstart?

When -Xquickstart is used, the JIT will perform compilations at a lower optimization level in order to compile as fast as possible (long term throughput may be degraded though). Moreover, the mechanism that gathers profiling information when methods run interpreted is turned off, which limits the ability of the optimizer further (think less inlining). If the optimizer is not doing a lot of work, the amount of memory it needs for its internal data structures is less.

And why is Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile not set automatically with the option -Xtune:virtualized?

-Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile looks at the amount of free physical memory at the beginning of a compilation and sets a limit for the scratch memory (the amount of memory a compilation thread is allowed to use) slightly less than the amount of physical memory. When a compilation tries to use more than its alloted limit, the JIT aborts that compilation and retries it at a lower optimization level. When the machine has swap space enabled, even though there is not enough physical memory, compilations can still use a lot of memory because of the swapping mechanism. We've found that -Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile, while safer, could make many compilations to be downgraded, resulting in lower throughput in the long run. This is why it is not enabled by default. Now, I agree that, when swap is disabled, we should probably enable -Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile to limit the amount of scratch memory based on the amount of physical memory. All I need is a way to detect from inside the JVM whether or not swap is enabled on the machine.

[Herr-Sepp]

Many thanks for the helpful explanations.

Hope that you find a way to implement a swap check, to make sure no one else gets OOM because of the JIT compilation anymore, when using K8s or similar container orchestrators.

[edrevo]

@mpirvu, is there any issue tracking the feature of automatically enabling -Xjit:enableSelfTuningScratchMemoryUsageBeforeCompile for non-swap enviroments?

[mpirvu]

I have created #9616 to track the requested feature. This issue can be closed now.