openj9 🚀 - Smooth migration story: mapping or supporting critical OpenJDK options

@gita-omr @vijaysun-omr @mstoodle @pshipton @DanHeidinga

zl-wang on 28 Feb 2019

@charliegracie @dmitripivkine @amicic

pshipton on 28 Feb 2019

Related Changes

| Description | PR | Issues |
| ------------- | ------------- | ------------- |
| -Xlp<size>, -Xlp:codecache:pagesize<size> order priority| https://github.com/eclipse/openj9/pull/7769 | |
| -Xlp<Size> Fix broken functionality | https://github.com/eclipse/openj9/pull/7722 | https://github.com/eclipse/openj9/issues/7667 |

AlenBadel on 3 Sep 2019

Just note that -Xlp format for object heap is -Xlp:objectheap:pagesize=<size>[,[non]pageble]]
where [non]pageble] is applicable for ZOS only.
Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

dmitripivkine on 3 Sep 2019

@dmitripivkine

Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

Does the JVM pull this from the OS defined large pages? Eg On Linux this is called hugepages which is set to 2M by default.

# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

If the user doubles Hugepagesize will the default value used by -Xlp (both objectheap and codecache) use that new value, or is the default value statically defined?
Only asking since this is the behaviour of XX:+UseLargePages

AlenBadel on 4 Sep 2019

@AlenBadel Do you know whether the -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache for Hotspot VMs? What about the failure cases where the sizes don't match?

We need to be careful that mapped options do the "right" thing on OpenJ9 that maps as closely as possible to the expected behaviour.

DanHeidinga on 4 Sep 2019

Do you know whether the -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache for Hotspot VMs?

Yes, on Hotspot VMs -XX:LargePageSizeInBytes=<size> effects both the heap and the code cache.

See

-XX:LargePageSizeInBytes=256m
Causes the Java heap, including the permanent generation, and the compiled code cache to use as a minimum size one 256 MB page (for those platforms which support it).
https://www.oracle.com/technetwork/java/tuning-139912.html

AlenBadel on 4 Sep 2019

👍1

XX:+UseLargePages can be easily mapped to -xlp.
However that changes in Java 9+, where -xlp is deprecated in exchange we need to enable both -Xlp:codecache and -Xlp:objectheap.

Here I'm assuming that -Xlp:codecache and -Xlp:objectheap will use OS defined large page sizes by default if no value is passed. That's what XX:+UseLargePages does.

-XX:LargePageSizeInBytes=<size> similarly can be mapped to -Xlp:codecache=<size> and
-Xlp:objectheap=<size>. Only issue here is that we don't have the infrastructure to map one to many, and that needs to be implemented.

-XX:ParallelCMSThreads=N was deprecated in JDK9+. -XconcurrentlevelN seems like the best candidate. Since it controls the number of concurrent GC threads for global marking in gencon.
Gencon is probably the best candidate GC to match CMS, along with another option to enable concurrent sweep.

@LinHu2016 Would you have any background on XconcurrentbackgroundN? I wasn't able to find any documentation on that option.

-XX:ConcGCThreads=N and -XX:ParallelGCThreads=N are closely related on Hotspot and are used to control their G1 GC. The relation by default is something like ConcGcThreads = (ParallelGCThreads + 3)/4.
I could only guess to use XconcurrentlevelN again here, but it's not obvious since it's not clear since these are G1 specific options.

Hi @AlenBadel, here is document link for -Xconcurrentbackground (https://www.eclipse.org/openj9/docs/xconcurrentbackground/), I still think -Xconcurrentbackground option is the most close to -XX:ConcGCThreads=N on hotspot(-XconcurrentlevelN is Allocation “tax” rate, in OpenJ9 we do have different implementation, which ask mutator threads to pay “tax” during concurrent mark)

LinHu2016 on 4 Sep 2019

Default behaviour might depend on platform. xLinux for instance would use 2M (large page) by default however pLinux default is 64k (small page) not 1g (large page)

Does the JVM pull this from the OS defined large pages? Eg On Linux this is called hugepages which is set to 2M by default.
# cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
If the user doubles Hugepagesize will the default value used by -Xlp (both objectheap and codecache) use that new value, or is the default value statically defined?
Only asking since this is the behaviour of XX:+UseLargePages

No. Preferred page sizes for objectheap are hardcoded https://github.com/eclipse/omr/blob/master/gc/base/GCExtensionsBase.cpp#L133
If preferred page size is not provisioned (known) in OS a smallest page size available would be selected (in case if there is no -Xlp request).
Preferred page sizes in many cases are selected because of support in OS/HW and could provide best performance like 64k on AIX or 1m pageable on ZOS.
At startup time Port Library structure (see omrvmem_supported_page_sizes()) is initialized with provisioned (known) page sizes in OS. Most of platforms would have one or two available sizes however there can be more than two. For example AIX can support [4k, 64k, 16m, 16g] or ZOS [4k_nonPageable, 1m_Pageable, 2g_nonpageable].
In most of the cases Large Pages are shared pages so using of them required extra provisioning (reservation) in OS. In case if memory allocation in LPs is requested but can not be fulfilled die lack of them in OS the warning would be issued an allocation would be attempted with smaller pages.
An importance of -Xlp options in any form on Linux platforms has been reduced significantly last years due Anonymous Huge Pages support. The customer can get Huge Pages benefits without any extra provisioning

dmitripivkine on 4 Sep 2019

I still think -Xconcurrentbackground option is the most close to -XX:ConcGCThreads=N on hotspot(-XconcurrentlevelN is Allocation “tax” rate, in OpenJ9 we do have different implementation, which ask mutator threads to pay “tax” during concurrent mark)

I agree. -Xconcurrentbackground should be used for both.

The objectheap then won't use the OS defined "Huge Pages"(Linux) unless we specify it using -Xlp:objectheap=< size of huge pages >.
To effectively map X:+UseLargePages we would need to immediately extract the size of the OS defined Huge Pages, or AIX, ZOS equivalent and map it to xlp:objectheap= < size of extracted huge pages>.

Need to figure out if this is the same case for the codecache.

AlenBadel on 4 Sep 2019

The objectheap then won't use the OS defined "Huge Pages"(Linux) unless we specify it using -Xlp:objectheap=< size of huge pages >.

except it is hardcoded 2m for xLinux or 1m for zLinux

dmitripivkine on 4 Sep 2019

except it is hardcoded 2m for xLinux or 1m for zLinux

@dmitripivkine
Even if we specify the size of the maximum page using -Xlp:objectheap=< size > ? I thought it was hard coded only the case that no size is specified.

AlenBadel on 4 Sep 2019

except it is hardcoded 2m for xLinux or 1m for zLinux

@dmitripivkine
Even if we specify the size of the maximum page using -Xlp:objectheap=< size > ? I thought it was hard coded only the case that no size is specified.

Default value would be overwritten by option if requested

dmitripivkine on 4 Sep 2019

Default value would be overwritten by option if requested

That was what I meant. My apologies if that wasn't clear.

AlenBadel on 4 Sep 2019

I've updated the changes and mapping required https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042.

AlenBadel on 5 Sep 2019

Would it be possible to summarize the default HotSpot and OpenJ9 behaviour for the pages sizes? For example:
1) How are the page sizes chosen when no XX:+UseLargePages or -Xlp is specified on the command line for HS and OpenJ9 correspondingly?
2) How are the page sizes selected when those options are specified in each VM correspondingly?
3) Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?
4) And just to confirm: specifying page size using -XX:LargePageSizeInBytes=\

gita-omr on 8 Sep 2019

How are the page sizes chosen when no XX:+UseLargePages or -Xlp is specified on the command line for HS and OpenJ9 correspondingly?

On Hotspot I don't have enough knowledge on how page sizes are chosen without specifying an option.
On OpenJ9 these values are hard coded and differ based on architecture.

As Dmitri commented:

Preferred page sizes for objectheap are hardcoded https://github.com/eclipse/omr/blob/master/gc/base/GCExtensionsBase.cpp#L133

How are the page sizes selected when those options are specified in each VM correspondingly?

As I've noted

XX:+UseLargePages
Sets the largest page file size used to be of what is set in the OS (or hypervisor, based on platform).

Xlp does exactly the same.

Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?

No, we're simply mapping XX:+UseLargePages to (Xlp: Java8) (Xlp:objectheap and Xlp:codecache Java 9+)

And just to confirm: specifying page size using -XX:LargePageSizeInBytes= or -Xlp: on the command line overrides all of the above.

-XX:LargePageSizeInBytes= overrides any hardcoded value, or default value with the size of the page passed in by the user.

AlenBadel on 9 Sep 2019

👍1

Are we going to change anything in terms of the actual page sizes chosen when XX:+UseLargePages is specified with OpenJ9?

No, we're simply mapping XX:+UseLargePages to (Xlp: Java8) (Xlp:objectheap and Xlp:codecache Java 9+)

Why do you need to do something for Java 8 differently? Java 8 and higher should have identical implementations

dmitripivkine on 9 Sep 2019

Why do you need to do something for Java 8 differently? Java 8 and higher should have identical implementations

This was a consideration due to the deprecation of -Xlp in Java9+.
https://www.eclipse.org/openj9/docs/xlp/

It would depend on the availability of -Xlp:codecache and -Xlp:objectheap. If they are available in Java 8, and the behaviour is the same then these options will be mapped instead of -Xlp.

AlenBadel on 9 Sep 2019

From my understanding -Xlp is deprecated, but is still accepted and is functionally sane in Java9+. Could anyone confirm this?

To avoid making changes to the mapping infrastructure, and implementing one to many mapping we could do one of two things.

If using -Xlp is functionally sane, then we can map -XX:+UseLargePages and -XX:LargePageSizeInBytes=<size> to -Xlp instead of both -Xlp:codecache, and -Xlp:opbjectheap. Which should be equivalent mappings.
This of course would need to be changed later on if we choose to remove -Xlp handling.

Keeping in mind that the codecache is much smaller than the objectheap. The performance yields would be higher using larger pages on an objectheap versus codecache. We can consider simply mapping -XX:+UseLargePages, and -XX:LargePageSizeInBytes=<size> to -xlp:objectheap. I don't see any potential issues with codecache using different page sizes as the objectheap.

AlenBadel on 9 Sep 2019

I believe this is movement in wrong direction. -Xlp functionality is not exactly the same as -Xlp:objectheap:* -Xlp:codecache:* for obsolete backwards compatibility issues. Also an implementation should not limit required functionality. So you can not say we would set -Xlp:objectheap: but not codecache because of limitations of implementation.

dmitripivkine on 9 Sep 2019

👍1

Doesn't seem anything is happening here for the 0.17 release, moving to the next one.

pshipton on 27 Sep 2019

Yeah, too much distraction by various defects...

gita-omr on 27 Sep 2019

A few things to update here.

Currently on Z the [non]pageable option needs to be passed in when using the following
-Xlp:codecache:pagesize=<size>,[non]pageable

The issue is that nonpageable large files are not supported on the Z codecache. I'll be making the change so that by default it uses pageable large files within the codecache. After this change, the codecache size on Z can be specified like all other platforms.
-Xlp:codecache:pagesize=<size>

A similar situation exists with the objectheap. An additional [non]pageable option is required.
-Xlp:objectheap:pagesize=<size>,[non]pageable

Since pageable large pages are preferred if both pageable, and nonpageable are available then we can do the same with the objectheap. This change will use pageable large files if available, otherwise will try using nonpageable large files. Again, the objectheap size on Z can be specified like all other platforms.
-Xlp:objectheap:pagesize=<size>

As we've mentioned -Xlp<size> attempts to force both the objectheap and codecache to use large pages of size. -Xlp attempts to use the largest available page files only within the objectheap. So there's a bit of inconsistency here, but I digress. We currently don't have a way to force the codecache to use the largest available on the system without specifying the size.

@pshipton
Would we benefit from allowing -Xlp:codecache and -Xlp:objectheap without passing the pagesize=<size> to allocate the largest large pages available on the system by default for the respective cache and heap?

I understand that -XX:+UseLargePages can be parsed directly inside the codecache and the behaviour can be unique to this option. Just wanted to make sure if J9 options can benefit from similar functionality.

AlenBadel on 21 Oct 2019

I don't see why we need to add additional OpenJ9 specific options if -XX:+UseLargePages provides the additional behavior needed.

pshipton on 21 Oct 2019

Although I don't have any particular objection to adding -Xlp:codecache and -Xlp:objectheap if somebody is asking for this kind of control.

pshipton on 21 Oct 2019

This change should NOT change current behaviour for -Xlp:objectheap:*

dmitripivkine on 21 Oct 2019

Although I don't have any particular objection to adding -Xlp:codecache and -Xlp:objectheap if somebody is asking for this kind of control.

Do you not agree that mapping should be just that, mapping? The functionality doesn't exist on OpenJ9 to automatically detect and use the largest page size to be used within the codecache. There's no reason in my opinion to introduce new functionality just to be used with a Hotspot option.

What you're suggesting is to parse -XX:+UseLargePages just as we parse -xlp:codecache: inside J9Options.cpp. See https://github.com/eclipse/openj9/blob/bc4b0b1be15b988c1c3fb3da295caa0d039c5eb5/runtime/compiler/control/J9Options.cpp#L1462
Aren't we trying to keep the reference of -XX:* Options all in one place?
See https://github.com/eclipse/openj9/blob/bc4b0b1be15b988c1c3fb3da295caa0d039c5eb5/runtime/vm/jvminit.c#L4110

Implementing the behaviour I've outlined above https://github.com/eclipse/openj9/issues/4930#issuecomment-544535848 would preserve the convention of keeping -XX* argument mappings together.

AlenBadel on 21 Oct 2019

This change should NOT change current behaviour for -Xlp:objectheap:*

Are you suggesting that we shouldn't change the current behaviour at all, or create another issue for this?

To apply symmetry if -Xlp:codecache uses the largest available large page size then it would be nice that-xlp:objectheap does the same.

AlenBadel on 21 Oct 2019

I think what @AlenBadel is saying is that introducting -Xlp:codecache and -Xlp:objectheap will help implement the mapping. This is because of the way mapping works right now: all -XX OpenJDK options are replaced with the string representation of the corresponding -X OpenJ9 options in a separate pass. Then, those -X options get processed.

Since -XX+UseLargePages does not have corresponding OpenJ9 options we have to skip it in the first pass and then parse it similar to the -Xlp option in the second pass (except for the way code cache page size is set)

gita-omr on 21 Oct 2019

-XX:+UseLargePages doesn't need to be a mapping. We don't need to think of it as a "Hotspot option". But anyway, I don't have any objection to -Xlp:codecache and -xlp:objectheap. Agree we need to have symmetry.

pshipton on 21 Oct 2019

Aren't we trying to keep the reference of -XX:* Options all in one place?

Not all -XX: options are mappings, so they aren't all parsed in one place. Some -XX: options are specific to OpenJ9, and some are common between Hotspot and OpenJ9.

pshipton on 21 Oct 2019

General guidance is that any new OpenJ9 options should use the form -XX: to be consistent with Hotspot.

pshipton on 21 Oct 2019

👍1

General guidance is that any new OpenJ9 options should use the form -XX: to be consistent with Hotspot.

Thanks for clearing that up.

If this is the case then I'll directly parse -XX:+UseLargePages rather than mapping it to reduce complexity.

AlenBadel on 21 Oct 2019

This change should NOT change current behaviour for -Xlp:objectheap:*

Are you suggesting that we shouldn't change the current behaviour at all, or create another issue for this?

To apply symmetry if -Xlp:codecache uses the largest available large page size then it would be nice that-xlp:objectheap does the same.

shouldn't change the current behaviour at all

dmitripivkine on 22 Oct 2019

There is my vision how handling for -XX:LargePageSizeInBytes=<size> implementation:

Do NOT use any mapping API for this option at ALL!
Parse option in VM code (where other -XX options are parsed) without CONTEXT - parameter should be unsigned decimal. Handle possible incorrect input. If parsed properly, store the numeric value as uintptr_t as well as POSITION of parameter in Java command line. Be sure you are handling MOST RIGTH -XX:LargePageSizeInBytes=<size> option in case there are more then one
Organize the way how to pass this uintptr_t to GC and JIT components along with information that desired page size is requested with -XX option and it's position.
Find correct place in GC and JIT code to integrate handling of passed numeric value with CONTEXT (for example do we have requested page size provisioned in OS? Do we have more then one kind of page of requested size? etc.). Provide Error / Warning messages if required. Check that option is not overwritten by related -Xlp*option (still be a rule most right wins).

We can extend existing OpenJ9 command line options if necessary. However changing of them just because to make direct "mapping"is easier is not what I would do. We have much better instruments to do the job. Please be accurate with existing logic and do not miss required steps for verification, reporting etc.

dmitripivkine on 22 Oct 2019

To apply symmetry if -Xlp:codecache uses the largest available large page size then it would be nice that-xlp:objectheap does the same.

AIX has largest available page size 16g and it is NOT we want to use as default

dmitripivkine on 22 Oct 2019

👍1

As an update.

There has been a number of work items that have come out of me looking at this issue.

(Unrelated) - Looking closely on the codecache, there was really no reason for ZOS required an extra pageable/nonpageable arguemnt since the codecache only supports pageable. So a change to address that behaviour was created.
See https://github.com/eclipse/openj9/pull/7585

I've moved mapping of -XX:ParallelCMSThreads=N, -XX:ConcGCThreads=N and -XX:ParallelGCThreads=N into a separate PR.
See https://github.com/eclipse/openj9/pull/7723

While cleaning up the options parsing code within J9Options.cpp I noticed this gcc7.3 bug on PPC64LE which was giving us the wrong index of the parsed option.
See Issue https://github.com/eclipse/openj9/issues/7667
Workaround PR: https://github.com/eclipse/openj9/pull/7722
Issue to address workaround in other places we're doing something similar
See Issue https://github.com/eclipse/openj9/issues/7746

AlenBadel on 12 Nov 2019

Taking into account @dmitripivkine comments https://github.com/eclipse/openj9/issues/4930#issuecomment-544753867

Parsing of -XX:+UseLargePages and -XX:LargePageSizeInBytes=<size>
As was mentioned. Parsing of the -XX options should be done in the same place. To date the parsing of the Hotspot related -XX mappable options has been implemented in _jvminit.c_. Particularly the parsing for these two options should be implemented in similar location and fashion to idle tuning. See https://github.com/eclipse/openj9/blob/e386a08034fbc03e3b67b01436bbb2c97f562d28/runtime/vm/jvminit.c#L1860

Next, we need to address what we're parsing and where we're storing them. As noted, we're going to be storing both the index of the right-most occurrence of either of these options. As well as storing the value of the large page size requested in bytes. This allows us to use two variables to define both options. Only one of these options can be used at once, and if they've both passed by the user only the option with the rightmost index will be used.

I propose adding the following members directly into the J9JavaVM struct. Since adding it into the vmRuntimeListener struct was deemed undesirable due to it being unrelated to the existing uses of that struct.

uintptr_t largePageSizeRequested;
IDATA largePageArgIndex;

This will allow us to define the following:
-XX:+UseLargePages
largePageSizeRequested = (index of option)
IDATA largePageArgIndex = -1 (Since the option lacks a size argument)

-XX:LargePageSizeInBytes= <size>
largePageSizeRequested = (index of option)
IDATA largePageArgIndex = (size)

Neither option used
largePageSizeRequested = -1
IDATA largePageArgIndex = -1

General Implementation
We need to establish these two options' behaviour with -Xlp:codecache:pagesize=<size>, -Xlp<Size>, and -Xlp. Currently, these options don't operate on index ordering.

If a user passes in java -Xlp:codecache:pagesize=16M -Xlp64K
The codecache will end up using 16M as the large page size rather than 64K even when the -Xlp option is at the right most index.

This is not the same behaviour seen within the objectheap. There is currently a check which compares the indices of both options, and ends up using the correct size of the rightmost option (64K)
java -Xlp:objectheap:pagesize=16M -Xlp64K
@pshipton Is this the expected behaviour?

Codecache Implementation
Assuming the above is the not correct behaviour. That all -Xlp indices are be treated equal, and priority is based on which has the right most index. Then we can simply expand this to our two new mapped options. That -XX:+UseLargePages will be used only if it is the right most option ahead of all other Xlp* options, and -XX:LargePageSizeInBytes. The same can be said for -XX:LargePageSizeInBytes.

-XX:LargePageSizeInBytes=<size> is very similar to logic used for -Xlp<size>. A lot of this code can be re-used with a bit of moving things around. I won't get into the specifics.

The same can't be said for -XX:+UseLargePages. However, we can easily use j9vmem_default_large_page_size_ex to get the default large page size without using context as it already accounts for special architecture requirements.

GC Implementation
A lot of the parsing of Xlp options is done here https://github.com/eclipse/openj9/blob/941845bc01193b2132141a7ff0bb8e61467f29c9/runtime/gc_modron_startup/mmparse.cpp#L570.

Ideally we would like to not introduce -XX:* option code to be hacked into a Xlp parsing method, or reproducing code to check for indices of Xlp options.
Keeping this modular, I propose that we change the return type of this method to an Integer. This allow us to return -1 in the case that there was an error during parsing, otherwise will return the index of the Xlp option used. The index returned will be that of the right most -Xlp option. This way, we can easily compare this value to the arguments we parsed earlier to override any large page size value set while parsing the -Xlp option with the values passed in by -XX:+UseLargePages or -XX:LargePageSizeInBytes.

AlenBadel on 12 Nov 2019

@pshipton @DanHeidinga @dmitripivkine please review Alen's design above.

gita-omr on 12 Nov 2019

@pshipton Is this the expected behaviour?

Yes it is the expected behavior for the right most option to control the behavior. The only time this wouldn't be strictly true is if the top level option supports combination of suboptions from multiple options. i.e. Say -Xjit:abc -Xjit:def was combined to -Xjit:abc,def.

While I don't think Parsing of the -XX options should be done in the same place. is strictly true, I don't have any objection to adding parsing of -XX:+UseLargePages and -XX:LargePageSizeInBytes=<size> into jvminit.c and adding the new variables to J9JavaVM. Dmitri can comment on changing the GC code.

pshipton on 13 Nov 2019

I assume Assuming the above is the not correct behaviour. should be Assuming the above is the correct behaviour. (with not removed).

pshipton on 13 Nov 2019

I assume Assuming the above is the not correct behaviour. should be Assuming the above is the correct behaviour. (with not removed).

Just to be clear, I was trying to point out a mismatch of behaviour between the codecache and the objectheap.

The codecache as it currently is implemented will NOT choose the right most option, while the objectheap is setup to.

I.e when running
java -Xlp:codecache:pagesize=16M -Xlp64K
codecache will use 16M rather than 64K. Hence will ignore the right-most option, and will always prioritize -Xlp:codecache:pagesize = above -Xlp. The codecache ignores the value of the index of each argument here.

java -Xlp:objectheap:pagesize=16M -Xlp64K
objectheap will use 64K. The objectheap checks the index of each of these options, and correctly uses the index of each value to determine which is the right-most option.

From what you've mentioned, it sounds like the codecache's behaviour isn't quite right and we need to be only considering the right most -Xlp option.

AlenBadel on 14 Nov 2019

it sounds like the codecache's behaviour isn't quite right and we need to be only considering the right most -Xlp option

This is correct.

pshipton on 14 Nov 2019

👍1

it sounds like the codecache's behaviour isn't quite right and we need to be only considering the right most -Xlp option

This is correct.

Great. Looks like we're in agreement. I'll go ahead and bring up a separate PR to correct this.

AlenBadel on 14 Nov 2019

GC Implementation
A lot of the parsing of Xlp options is done here

https://github.com/eclipse/openj9/blob/941845bc01193b2132141a7ff0bb8e61467f29c9/runtime/gc_modron_startup/mmparse.cpp#L570

.
Ideally we would like to not introduce -XX:* option code to be hacked into a Xlp parsing method, or reproducing code to check for indices of Xlp options.
Keeping this modular, I propose that we change the return type of this method to an Integer. This allow us to return -1 in the case that there was an error during parsing, otherwise will return the index of the Xlp option used. The index returned will be that of the right most -Xlp option. This way, we can easily compare this value to the arguments we parsed earlier to override any large page size value set while parsing the -Xlp option with the values passed in by -XX:+UseLargePages or -XX:LargePageSizeInBytes.

You can do this. The beneficial part of this approach you would not change existing -Xlp behaviour. However it means you will need to add for new options check for availability of the requested page size in OS, choose "default large page" if size is not requested, choose missed value for [non]pageable for ZOS and proper error/warning reporting/handling

dmitripivkine on 15 Nov 2019

However it means you will need to add for new options check for availability of the requested page size in OS, choose "default large page" if size is not requested, choose missed value for [non]pageable for ZOS and proper error/warning reporting/handling

Right, which still can be done effectively since a lot of these checks are common to both Xlp options and our new mapped options. I can throw the common code into a function that is called in both cases which means the codes wouldn't need to be redundantly implemented.

AlenBadel on 16 Nov 2019

I must also note that it's a requirement to add tests for both -XX:+UseLargePages and -XX:LargePageSizeInBytes=<size>.

AlenBadel on 16 Nov 2019

I've went ahead and added more tests to our xlp codecache test cases. These tests target the behaviour of what large page file size is selected under multiple different combinations of xlp user augments. The same tests already exist for the objectheap.

See:
https://github.com/AlenBadel/openj9/commit/0f0e5658358fce0c7b347ba65f3d1994614d0705

It's yet to be tested but requires https://github.com/eclipse/openj9/pull/7769 to be merged to be functionally sane.

AlenBadel on 2 Dec 2019

Updated the design see https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 .

AlenBadel on 17 Dec 2019

Moving out of 0.18 as we're long past the code split

DanHeidinga on 7 Jan 2020

I think what is happening is that @AlenBadel keeps creating PRs and all the discussion happens in those PRs. Sometimes those discussions lead to a complete redesign.
Instead, I would like to treat https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 as a design document (the table) and finalize it first. @dmitripivkine @zl-wang @joransiu @DanHeidinga could you please review the table and comment/approve?

gita-omr on 9 Jan 2020

I think what is happening is that @AlenBadel keeps creating PRs and all the discussion happens in those PRs. Sometimes those discussions lead to a complete redesign.
Instead, I would like to treat #4930 (comment) as a design document (the table) and finalize it first. @dmitripivkine @zl-wang @joransiu @DanHeidinga could you please review the table and comment/approve?

I believe we do agree on behaviour for options to be added. However there is problem with implementation and it is why I am telling pre-design cleanup required. And for problematic ZOS case your suggestion would not help. Description is not specific enough.

So, again, for ZOS 64 bit for object heap:

page size/type might be discovered (it is not hardcoded but typical):

4k fixed
1m fixed
1m pageable
2g fixed

Desired page size for object heap: 1M pageable

-Xlp (size not specified) 1m fixed
-Xlp<size> if size is 1m use 1m fixed first
-Xlp:objectheap:pagesize=<size>,[non]pageable fully specified page request, no default

now for new options:

-XX:+UseLargePages use 1m pageable (BTW it is a default any way)
-XX:LargePageSizeInBytes=<size> if size is 1m use 1m pageable first

So it is obvious that -XX:+UseLargePages can not be mapped to -Xlp because of different desired behaviour. And an attempt to infuse new option to existing code is hard - either we need to hack default values for new options (silly) or change behaviour for old options to be compatible.

If we can change prioritization for -Xlp and -Xlp<size> to get 1m pageable first it would help a lot with this PR. I don't have objections but I would like to have experts approval @fjeremic @joransiu
I have no idea how many customers might be impacted because of such change.

If we can not change current defaults we need to integrate new duplicating (but different output) API to Port for ZOS code. Again, looks silly.

dmitripivkine on 10 Jan 2020

To be particular a question that needs to be addressed is whether to prefer pageable or fixed large pages on Z/OS. @joransiu Mentioned that pageable would be preferred, however wouldn't fixed large pages be more performance advantageous? Would we be creating a performance degradation switching the default behaviour for -Xlp, and -Xlp<size>?

AlenBadel on 10 Jan 2020

We also need to address what the exact behaviour of -XX:+UseLargePages is on HotSpot. My only concern is if there is an upper limit to what large pages are utilized. The official documentation doesn't really answer that. Perhaps @pshipton @DanHeidinga would be able to help answer my question.

Using Z/OS as an example, if we have 4k/1M/2G large files configured will -XX:+UseLargePages on Hotspot use 2G large files? Will it be limited by the heap size if specified by the user? The lack of clarification on the upper bound of large pages to be used leaves me with the idea that it's boundless and it will use the largest available.

I'm looking into configuring a machine with multiple large page sizes, and see if I can answer that question experimentally.

AlenBadel on 10 Jan 2020

In regards to https://github.com/eclipse/openj9/issues/4930#issuecomment-573167155.

I think the suitable approach will be to follow the current behaviour and convention used within -Xlp. The current behaviour states that nonpageable large pages will be used by default on ZOS. Since nonpageable, or fixed large pages hold performance benefits over pageable large pages. I believe it is best to also extend this behaviour to both -XX:+UseLargePages, & -XX:LargePageSizeInBytes=<size>.

It is still not known what the upper bound of configured large pages -XX:+UseLargePages will utilize on HotSpot. Nevertheless, it's meaningless to select large pages that may be greater than the heap the Java program will utilize. I believe we should again continue to extend the behaviour of -Xlp, since each platform already has a list of preferred large pages that will result in a sizable performance increase.

@dmitripivkine Please let me know if you have any further concerns.

AlenBadel on 14 Jan 2020

In regards to #4930 (comment).

I think the suitable approach will be to follow the current behaviour and convention used within -Xlp. The current behaviour states that nonpageable large pages will be used by default on ZOS. Since nonpageable, or fixed large pages hold performance benefits over pageable large pages. I believe it is best to also extend this behaviour to both -XX:+UseLargePages, & -XX:LargePageSizeInBytes=<size>.

It is still not known what the upper bound of configured large pages -XX:+UseLargePages will utilize on HotSpot. Nevertheless, it's meaningless to select large pages that may be greater than the heap the Java program will utilize. I believe we should again continue to extend the behaviour of -Xlp, since each platform already has a list of preferred large pages that will result in a sizable performance increase.

@dmitripivkine Please let me know if you have any further concerns.

For -XX:+UseLargePages I have no concerns however this decision also needs to be approved by ZOS specialists @fjeremic @joransiu Using of fixed not pageable it is old old -Xlp behaviour and It is not what I heard from them before (I believe last advice was to prioritize using of pageable). Also if changed way you suggest and considering 1m pageable is a default in case no -Xlp specified so using of -XX:+UseLargePages would make situation worse.

For -XX:LargePageSizeInBytes=<size> I believe this is it not a right approach. As far as you need to guess type of the page any way why not to prioritize pageable over fixed?

dmitripivkine on 15 Jan 2020

As far as you need to guess type of the page any way why not to prioritize pageable over fixed?

It's not clear what the benefits pageable has over fixed. There's a reason why currently -Xlp assumes fixed over pageable, and that is due to the performance benefits.

AlenBadel on 15 Jan 2020

As far as you need to guess type of the page any way why not to prioritize pageable over fixed?

It's not clear what the benefits pageable has over fixed. There's a reason why currently -Xlp assumes fixed over pageable, and that is due to the performance benefits.

pageable pages can be assigned dynamically run time by ZOS Memory Manager. It is ZOS equivalent of virtual memory reservation on demand Linux support. Using pageable brings more flexibility to distribute memory resources between applications. fixed pages are statically pinned to application and can not be re-destributed on OS level. Even if fixed pages are better performance wise pageable are beneficial for system obviously.

dmitripivkine on 15 Jan 2020

Even if fixed pages are better performance wise pageable are beneficial for system obviously.

Understood. I'll leave that decision up to S390 experts as to which should be used by default.

AlenBadel on 15 Jan 2020

From a user consumability perspective, pageable large pages is easier to manage. fixed requires z/OS users to have pre-reserved an amount real memory at IPL time dedicated to back up the fixed pages. From a performance persective, I'd expect the address translation savings to be quite similar.

I suspect we kept fixed pages as default for -Xlp because initially, only fixed 1MB pages were available on z/OS. pageable large pages came in a subsquent z/OS release. We likely wanted to avoid changing the default behaviour, in case, users had pre-allocated the real storage for fixed, and are very careful in terms of the order in bringing up their applications that would consume these pages. Honestly, I don't know how much fixed pages are used in practice nowadays, relative to pageable. I'll follow-up with z/OS RSM folks to see if they have further insights on usage.

joransiu on 15 Jan 2020

👍1

If we decide that XX:UseLargePages should directly map into -Xlp (old or modified) we don't need to reverse engineer it's current behavior in HotSpot.

gita-omr on 16 Jan 2020

Based on advice from z/OS RSM folks we've decided to keep using fixed large pages within -Xlp, and -Xlp<size>.
The major reason for this decision is that Z/OS can not guarantee that pageable large pages can be backed with 1M pages, and the large page can be degraded to be backed with 4K pages. Currently, users expect that their applications will always use 1M large pages, and this behaviour is critical to their application.

Moving forward, -XX:+UseLargePages, and -XX:LargePageSizeInBytes=<size> will prefer pageable large pages on Z/OS.

AlenBadel on 21 Jan 2020

While attempting to enable the xlpCodecache tests on AIX, I noticed we are printing the incorrect list of page sizes available.

  -Xlp:objectheap:pagesize=64K   large page size
                  available large page sizes:
                  4K
                  64K.  <--- 64K is listed.
                  16M
                  16G
  -Xlp:codecache:pagesize=64K    large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  4K.  
                  16M.   <-- Missing 64K, even when 64K is being used to back the code cache.

This is causing the test to fail on AIX. Hoping to resolve this issue.

AlenBadel on 23 Jan 2020

I've raised an issue to track https://github.com/eclipse/openj9/issues/4930#issuecomment-577785110.

See: https://github.com/eclipse/openj9/issues/8437

AlenBadel on 28 Jan 2020

Reviewing the design above:

XX:LargePageSizeInBytes= | Ignored | Set large page size specified by user. (Z/OS) Will use pageable large page with specified . If that large page is not configured, will use the next closest default large page size. Preferring pageable large pages, as in -XX:+UseLargePages.
-- | -- | --

1) What does it mean "next closest default"?
2) What if the alternative is not available?
3) Last sentence is not complete.

gita-omr on 31 Jan 2020

(Z/OS) -Xlp | Objectheap chooses the largest default fixed large page configured on the system up to 1M. The codecache ignores this option.
-- | --
1) is there more than one default out of which we choose largest? Did you mean "supported by HW and OS" ?
2) similar with -Xlp=_size_

gita-omr on 31 Jan 2020

(Z/OS) -Xlp Objectheap chooses the largest default fixed large page configured on the system up to 1M. The codecache ignores this option.

is there more than one default out of which we choose largest? Did you mean "supported by HW and OS" ?

similar with -Xlp=_size_

right. And what means "largest"?

dmitripivkine on 31 Jan 2020

What does it mean "next closest default"?

Z/OS has the following available page sizes that could be configured.
[2G Fixed, 1M Pageable, 1M Fixed, 4K Fixed]

If the specified page size is not available, then the greatest available page size below that value will be used.

Let's consider the following scenario. The user wants to specify-XX:LargePageSizeInBytes=2G, and the following pages are configured [1M Fixed, 1M Pageable, 4K Fixed] on the system. 1M Pageable will be used. Since 1M pages are the next closest value to the one desired, and we're preferring pageable over fixed.

If we change our scenario so that 1M pageable is not configured, then 1M Fixed will be used.

What if the alternative is not available?

We'll be going down the chain, finding the greatest page size value under the specified value. If no pages are configured, we'll end up using 4K fixed pages.

is there more than one default out of which we choose largest? Did you mean "supported by HW and OS"

Yes, whatever is supported by the hardware. Keeping in mind that codecache does not support fixed large pages.

AlenBadel on 10 Feb 2020

@AlenBadel please update the table in the design to be more specific.

gita-omr on 10 Feb 2020

👍1

The design above has been updated to include the specifics I've noted yesterday.

AlenBadel on 11 Feb 2020

@dmitripivkine @gita-omr Please let me know if you have any further concerns.

@joransiu Could you kindly review the Z/OS behaviour in https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 and just give me a simple approval if you have no further questions. The Z/OS relevant change can be seen https://github.com/eclipse/omr/pull/4762.

AlenBadel on 11 Feb 2020

Design at https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 looks good to me.

gita-omr on 11 Feb 2020

As I've noted above, this issue is waiting closing feedback from @dmitripivkine and @joransiu.

AlenBadel on 12 Feb 2020

XX:+UseLargePages

The Objectheap and Codecache will use the largest available large page size configured on the system.

This statement is not correct. "Largest available large page size" on ZOS might be 2G and on AIX 16G.
These sizes are not going to be used

dmitripivkine on 12 Feb 2020

Xlp on ZOS

Currently, ZOS requires an additional argument(pageable/nonpageable) when specifying the large page size to be used within the ObjectHeap or CodeCache.

-Xlp:codecache:pagesize=,[non]pageable
-Xlp:objectheap:pagesize=,[non]pageable
It is the only platform to have this requirement. Nonpageable large pages is currently not supported within the codecache. Pageable large pages are also desired over NonPageable large pages within the ObjectHeap.

It would be desirable to have a uniform option that works across all platforms.

PR: #7585

We are NOT going to do this

dmitripivkine on 12 Feb 2020

We have special behaviour on AIX as well (similar as on ZOS). Did you look to it?

dmitripivkine on 12 Feb 2020

PR: #7585

We are NOT going to do this

I think it would be fine to keep this text but say it will be discussed further in PR: 7585

gita-omr on 12 Feb 2020

We have special behaviour on AIX as well (similar as on ZOS). Did you look to it?

@dmitripivkine do you think it will affect the mapping of the XX options? I think we only need to discuss the aspects that affect the mapping.

gita-omr on 12 Feb 2020

XX:+UseLargePages

The Objectheap and Codecache will use the largest available large page size configured on the system.

This statement is not correct. "Largest available large page size" on ZOS might be 2G and on AIX 16G.
These sizes are not going to be used

Well the whole sentence states

The Objectheap and Codecache will use the largest available large page size configured on the system. Each platform has certain upper limits to what large page size can be used, and for these specs please take a look at the limits specified in the xlp docs.

I've change it to:

The Objectheap and Codecache will use the largest available large page size configured up to a defined ceiling. Each platform has certain upper limits to what large page size can be used, and for these specs please take a look at the limits specified in the xlp docs.

The upper limit is already defined in the Xlp docs. Eg 1M on Z/OS and 16M on PPCLE.

AlenBadel on 13 Feb 2020

👍1

We have special behaviour on AIX as well (similar as on ZOS). Did you look to it?

I have looked into this. The size determination algorithm inside the port library which determines which page sizes should be used will stay unchanged for these options. We're respecting and expecting the same behaviour as seen using -Xlp on AIX. The only difference is that with -XX:+UseLargePages both the codecache and objectheap will use large pages, compared to Xlp where codecache ignored such action.

AlenBadel on 13 Feb 2020

👍1

PR: #7585
We are NOT going to do this

I think it would be fine to keep this text but say it will be discussed further in PR: 7585

Right. I'll spawn a related issue for this.

AlenBadel on 13 Feb 2020

👍1

We have special behaviour on AIX as well (similar as on ZOS). Did you look to it?

@dmitripivkine do you think it will affect the mapping of the XX options? I think we only need to discuss the aspects that affect the mapping.

AIX implementation for -Xlp has own exceptions. For example -Xlp would push to use 16m pages.
If it is the behaviour we want to have for XX:+UseLargePages as well so no problem.
Another exception default large page size for executable is selected (as comment in the code state) "same page size as used by JIT code cache when allocated using omrmem_allocate_memory()" - just FYI

dmitripivkine on 13 Feb 2020

Another exception default large page size for executable is selected (as comment in the code state) "same page size as used by JIT code cache when allocated using omrmem_allocate_memory()

By default, the OpenJ9 launcher sets AIX to use 64K pages. When the user passes in -XX:+UseLargePages this will set the codecache to use 64K pages and this should always return 64K. AIX does not effectively use 16M pages, so to avoid adverse performance effects we'll stay to always using 64K unless the user specifies using 16M explicitly.

If the user specifies 16M explicitly, it will end up using 16M if it's configured there's no problem with that path.

AlenBadel on 13 Feb 2020

Please try your best to avoid misinformation:

By default, the OpenJ9 launcher sets AIX to use 64K pages.

Is this confirmed? see this issue I opened yesterday: https://github.com/eclipse/openj9/issues/8562

AIX does not effectively use 16M pages

Binary size reality is the reason that 16M pages might not provide as big a benefit as expected. Plus, IERAT on some hardware(s) doesn't support 16MB pages. Further reducing its effect. But it is not AIX's reason.

zl-wang on 13 Feb 2020

Is this confirmed?

Not yet, as designed that's the intention. I'll link the issue you've created https://github.com/eclipse/openj9/issues/8562. If that's broken, and that it's the case that 64K pages are not being used by default then it'll be fixed to make sure that 64K pages are used by default.

AlenBadel on 13 Feb 2020

I think this discussion is moving in a wrong direction again. I don't think the purpose of this issue is to fix/document everything about the current -Xlp behavior. I think all the design at https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 should say is that -XX:UseLargePages will select page size and type in exactly the same way as -Xlp, except for zOS. Then, describe the zOS behavior. I think it was written that way at some point but then got lost after editing. Also I believe zOS is not supported by OpenJ9 so it's a bit sad that this issue is being delayed so much due to it.

I also recommend to remove all the text in https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042 after the table since a lot of it is a duplication and only causes more confusion.

And, of course, all the discovered bugs and potential improvements should be addressed in separate issues.

gita-omr on 14 Feb 2020

👍1

@dmitripivkine Does https://github.com/eclipse/openj9/issues/4930#issuecomment-585935235 answer your remaining concerns?

@joransiu Just a reminder that I need your OK on Z/OS relevant proposals here.

AlenBadel on 14 Feb 2020

I've went ahead and cleaned this issue up a bit. There was a lot going on.

I've created two children issues to address the options in two different groups. They're separated as general GC options, and large page related options.
The table of contents can be seen here: https://github.com/eclipse/openj9/issues/4930#issuecomment-527623042

AlenBadel on 26 Feb 2020

Seems this won't be resolved in the 0.20 release, moving it forward.

pshipton on 18 Mar 2020

As this wasn't resolved before branching for the 0.21 release, moving to the next.

pshipton on 9 Jun 2020

We are trying to add similar (possibly identical) functionality to OpenJDK's -XX:+AlwaysPreTouch here: https://github.com/eclipse/openj9/pull/10386. Anything else we should do, if we match the option name?

amicic on 13 Aug 2020

We are trying to add similar (possibly identical) functionality to OpenJDK's -XX:+AlwaysPreTouch here: #10386. Anything else we should do, if we match the option name?

We should reuse the existing OpenJDK option and avoid introducing a similar feature under a different name. In general, all new options should be of the form -XX:[+-]Something or -XX:Something.

DanHeidinga on 13 Aug 2020

moved to next milestone

dmitripivkine on 2 Sep 2020

Openj9: Smooth migration story: mapping or supporting critical OpenJDK options

All 94 comments

Table of Contents

Related issues