Openj9: -Xsoftmx doesn't appear to work as expected

[pwagland]

Java -version output

# java -version
openjdk version "11.0.9" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.23.0, JRE 11 Linux amd64-64-Bit Compressed References 20201022_810 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 3b09cfd7e9 based on jdk-11.0.9+11)

Summary of problem

Given my understanding of the documentation -Xsoftmx=600M shoaled mean that the JVM won't use more than 600M of heap. This can be tuned at runtime to go above this limit if needed.

The documentation at https://www.eclipse.org/openj9/docs/xsoftmx/ is perhaps ambiguous, since it _also_ states that:

There is no guarantee that a decrease in the heap size will occur. If or when the heap shrinks to less than the new limit, the heap will not grow beyond that limit.

It is unclear if it should go above the limit if it has never been above the limit.

In any case, in my test, I am allocating many 1M-1.5M objects, and then keeping ten of these in memory. This is creating a fair amount of garbage ~1.25M objects. I have specified -Xsoftmx=640M and so I wouldn't expect that the JVM should allocate more than 600M of heap, but I see it going to above 1G of heap.

Diagnostic files

I have attached my gc logs (very slightly sanitised to remove company names and directory locations from the program args)
mem.output.conf4-1109-options-0.2020-11-06T160657.gclog.zip

[pshipton]

@dmitripivkine pls take a look.

[dmitripivkine]

@jason-hall Would you please take a look?

[dmitripivkine]

@pwagland Thank you for using OpenJ9! Your expectations are correct - object heap should not be expanded over provided limit. We are going to investigate. Thank you very much to let us know.

[pmhayward]

If this will require a doc update, please add doc:externals label and raise an issue at github.com/eclipse/openj9-docs.

[dmitripivkine]

Current -Xsoftmx feature behaviour is ignoring watermark limit at initial heap expansion. Later on however GC is trying to contract heap below watermark. Once achieved the OOM is thrown in case if level is crossed it up again.
I can try older JVMs to verify is this behaviour exist since feature introduction or was broken some time later.

Our test https://github.com/eclipse/openj9/blob/master/test/functional/JLM_Tests/src/j9vm/test/softmx/SoftmxAdvanceTest.java verifies GC contraction behaviour as well as usage of MX Beans to change Softmx level dynamically.

We can change behaviour to respect watermark level at initial expansion. However it might be a reason for failures for customer loads expect heap expansion over watermark at application establishing time.

So choice is change behaviour and make it straight forward (with risk to break existing users) or preserve current behaviour and correct documentation accordingly.

@pshipton @tajila @manqingl What do you think?

[pshipton]

+1 for fixing it to respect the soft limit during the initial heap expansion. The option seems less useful if the soft limit is ignored. The heap may never contract below the watermark.

After the initial heap expansion, if it's expanded again does any OOM occur? If not, the soft limit is being totally ignored.

If we think the current behavior is useful, I'm not sure it is, another option could be added -XX:+IgnoreSoftMxForInitialExpansion.