Openj9: Supporting Hotspot Large Page Options

[AlenBadel]

Supporting Hotspot Large Page Options

Goals

1. To allow ease of migration from OpenJDK + Hotspot to OpenJDK + OpenJ9, this activity will focus implementing support for Hotspot large page options on OpenJ9

2. There are a number of inconsistencies with the existing large page -Xlp options (-Xlp, -Xlp<size>, -Xlp:objectheap, and -Xlp:codecache) this issue will resolve these inconsistencies. These inconsistencies will be resolved by:

   • Identifying and fix current Issues with -Xlp options.
   • Deprecate the existing -Xlp large page options.
   • Implement new paging options that are backwards compatible with -Xlp options allowing the -Xlp options to be mapped to the new options until their removal, while preserving legacy behavior.

Parent Issue: https://github.com/eclipse/openj9/issues/4930

Design

Design Doc: https://ibm.box.com/s/rys4fllji6ji0h1a6tvw79j16w8cum5g

Implementation

Parsing

Currently, all -Xlp options parsing is done within the CodeCache[1], and ObjectHeap[2] somewhat redundantly. This needs to be removed in favor for a centralized solution, where the options are parsed within the VM rather than each component that they affect. Each component should only need to read the parsed results within shared data structure, and configure large pages. Thus the parsing code in [1], and [2] will be removed in favor for a centralized VM implementation.

Centralized Parsing

The solution to this issue will be centralized parsing. The VM will parse all large command line arguments(-XX, & -Xlp options) within jvminit.c[3], and save the net result into shared data structure. The net parsing result consists of the combined affect of all active large page options, neglecting those that are overwritten due to precedence rules. The parsing result will be stored within a new struct inside J9JavaVM[4]. We can name this structure J9LargePageOptionsInfo.

J9LargePageOptionsInfo will consist of the following:

BOOLEAN isEnabledForCodeCache /* Enabled LP on CodeCache */
BOOLEAN isEnabledForObjectHeap /* Enabled LP on ObjectHeap */
UDATA pageSizeForCodeCache /* Specified LP size on CodeCache */
UDATA pageSizeForObjectHeap  /* Specified LP size on ObjectHeap */
J9LargePageCautionLevel lpCautionLevel /* Specify if Warnings, Errors, or Neither have been enabled */
UDATA pageTypeForObjectHeap /* Z/OS Only: Specify if user desires a specific page type on ObjectHeap*/

Where J9LargePageCautionLevel is an enum to specify the following states:

J9CautionUnset /* User has not enabled Warnings, or Errors */
J9CautionWarning /* Enabled Warnings */
J9CautionError /* Enabled Errors */

and pageTypeObjectHeap can be the following:

J9PORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE /* User does not specify to explicitly use a page type. */
J9PORT_VMEM_PAGE_FLAG_PAGEABLE /* User explicitly asks for pageable large pages */
J9PORT_VMEM_PAGE_FLAG_FIXED /* User explicitly asks for fixed large pages */

Populating J9LargePageOptionsInfo

Specifically, this section will focus on the implementation of parsing all -XX, and -Xlp large page options within jvminit.c[3].
This is an area of discussion, since we have two possible implementations.

Firstly, to understand what options are passed in and their position among the user arguments each option variation will need to be searched and consumed[10]. The macro FIND_AND_CONSUME will return the right-most index of an option and if a match is not found the index will return -1. The option will also be consumed if it is found. FIND_AND_CONSUME will be called for each option that is being supported, including all newly proposed -XX options and -Xlp options.

Since each -XX option relates to only a specific subset of J9LargePageOptionsInfo, the implementation can be partitioned to focus on one subset at a time.

An example of this to analyze all options that enable, or disable large pages. Namely these are -XX:[+/-]UseLargePages, -XX:[+/-]UseLargePagesCodeCache, -XX:[+/-]UseLargePagesObjectHeap, -Xlp(-Xlp, -Xlp<size>, -Xlp:[codecache/objectheap].

/* -XX:[-/+]UseLargePages[/CodeCache/ObjectHeap] */
{
    IDATA argUseLargePagesEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePages', NULL);
    IDATA argUseLargePagesDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePages', NULL);
    IDATA argUseLargePagesCodeCacheEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePagesCodeCache', NULL);
    IDATA argUseLargePagesCodeCacheDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePagesCodeCache', NULL);
    IDATA argUseLargePagesObjectHeapEnable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:+UseLargePagesObjectHeap', NULL);
    IDATA argUseLargePagesObjectHeapDisable = FIND_AND_CONSUME_ARG(EXACT_MATCH, '-XX:-UseLargePagesObjectHeap', NULL);

    /* Simplify to Codecache, and ObjectHeap components */
    argUseLargePagesCodeCacheEnable = OMR_MAX(argUseLargePagesCodeCacheEnable, argUseLargePagesEnable);
    argUseLargePagesCodeCacheDisable = OMR_MAX(argUseLargePagesCodeCacheDisable, argUseLargePagesDisable);
    argUseLargePagesObjectHeapEnable = OMR_MAX(argUseLargePagesObjectHeapEnable, argUseLargePagesEnable);
    argUseLargePagesObjectHeapDisable = OMR_MAX(argUseLargePagesObjectHeapDisable, argUseLargePagesDisable);

    /* Apply -Xlp mapping */
    argUseLargePagesCodeCacheEnable = OMR_MAX(argUseLargePagesCodeCacheEnable, argXlpEnableLargePagesCodeCache);
    argUseLargePagesObjectHeapEnable = OMR_MAX(argUseLargePagesObjectHeapEnable, argXlpEnableLargePagesObjectHeap);

    /* Set Lp Status Flags */
    lpInfo->isEnabledForCodeCache = argUseLargePagesCodeCacheEnable > argUseLargePagesCodeCacheDisable;
    lpInfo->isEnabledForObjectHeap = argUseLargePagesObjectHeapEnable > argUseLargePagesObjectHeapDisable;
}

Parsing -Xlp options
A contrary to the preceding -Xlp ruins the party, since it may impact all members at the same time. Hence it's best to parse -Xlp, -Xlp<size>, -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn],[pageable/nonpageable] ahead of parsing -XX options.

Xlp options will be parsed together. A set of front-facing (-XX large page compatabile) arguments can be used to translate and integrate the xlp options into how we parse the -XX large page options. The Xlp options can be parsed and simplified to the following variables.

IDATA argXlpEnableLargePagesCodeCache = -1;  // Index of right-most Xlp option that enables large pages on the codecache
IDATA argXlpEnableLargePagesObjectHeap = -1; // Index of right-most Xlp option that enables large pages on the objectheap

IDATA argXlpLargePageSizeInBytesCodeCache = -1; // Index of right-most Xlp option specifies a large page size for the codecache
UDATA xlpLargePageSizeCodeCache = -1;           // Value of that page size
IDATA argXlpLargePageSizeInBytesObjectHeap = -1; // Index of right-most Xlp option specifies a large page size for the objectheap
UDATA xlpLargePageSizeObjectHeap = -1;          // Value of that page size

IDATA argXlpPageWarningsEnable = -1;            // Index of right-most Xlp option specifies a "warn"
IDATA argXlpPageErrorsEnable = -1;            // Index of right-most Xlp option specifies a "strict"

#if defined(J9ZOS390)
IDATA argXlpObjectHeapPageType = -1;            // Index of right-most Xlp option specifies a [non]pageable
IDATA xlpObjectHeapPageType = -1;               // The value of non[pageable]
#endif

Specifics of Xlp, and -Xlp<size> are parsed using straight forward memory mapping.
-Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn],[pageable/nonpageable] will need to be parsed by a sub option parser. Thankfully one already exists within the GC[9]. Since -Xlp parsing can be removed from the GC, the xlpSubOptionsParser can be moved to jvminit.c and used to parse any combination of this option.

Similarly, there will be isolated code to handle and populate other members of this instance. Including -XX:LargePageInBytes[/CodeCache/ObjectHeap]=<size>, -XX:[+/-]LargePageWarnings and -XX:[+/-]LargePageErrors, and -XX:zOSLargePagesObjectHeapType=<type>

isEnabledForCodeCache
This member will be enabled only when an option that enables large pages on the codecache is either on the right of an option to disable it, or in absence of such option. As shown in the example above, we know the indices of each option variation this can easily be calculated.

isEnabledForObjectHeap
The Above can be applied for the ObjectHeap as well.

pageSizeForCodeCache
This will be set in accordance with the size specified on the right most index of any option that impacts the codecache page size. Namely, these are -Xlp<Size>, -Xlp:codecache:pagesize<size>, -XX:LargePageInBytes=<size> and -XX:LargePageInBytesCodeCache=<size>.

pageSizeForObjectHeap
The Above can be applied for the ObjectHeap as well. Namely, the options that impact the objectheap pagesize are -Xlp<Size>, -Xlp:codecache:pagesize<size>, -XX:LargePageInBytes<size> and -XX:LargePageInBytesObjectHeap=<size>.

lpCautionLevel
Warnings, will be enabled if options to enable it are to the right or in the absence of any option to disable it. The same applies to Errors.
Options that enable Warnings include: -XX:+LargePageWarnings, -Xlp:[codecache/objectheap]:pagesize=<size>,warn.
Options that disable Warnings include: -XX:-LargePageWarnings

Options that enable Errors include: -XX:+LargePageErrors, -Xlp:[codecache/objectheap]:pagesize=<size>,strict
Options that disable Errors include: -XX:-LargePageErrors.

(Z/OS only) pageTypeForObjectHeap
As, before in the absence of any option to specify the page type this will be set to J9PORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE, which will attempt for pageable variations but will fall back to fixed large pages. Otherwise, if an option that specifies the pagetype (-XX:zOSLargePagesObjectHeap=[pageable/nonpageable], or -XX:objectheap:pagesize=<size>,[non]pageable) exists the page type specified by the right-most index will be saved. (Pageable - J9PORT_VMEM_PAGE_FLAG_PAGEABLE, NonPageable - J9PORT_VMEM_PAGE_FLAG_FIXED).

(Common) Post-Parsing Implementation Details

In both options, the following checks will be necessary:

• When Xlp is detected a deprecation warning will be shown.
• When warnings, or errors are specified notify if large pages are enabled without specifying a large page size.

CodeCache Implementation

All Parsing will be removed from the codecache.[1]
Removing unused Xlp JIT .nls macros. Modify, and move to VM .nls for use within the VM.
Instead, large page configuration variables will be read from J9LargePageOptionsInfo.[5]
How the codecache verifies, and configures the large pages will not change. [6]

ObjectHeap Implementation

All Parsing will be removed from the objectheap. [2]
Removing unused Xlp GC .nls macros.
Instead, large page configuration variables will be read from J9LargePAgeOptionsInfo. [7]
How the objectheap verifies, and configures the large pages will not change, other than to accompany using data extracted from J9LargePAgeOptionsInfo. [8]

Related

An undocumented -Xlp option that manages the gc meta data page size is to be removed. -Xlp:gcmetadata will not be deprecated, rather it will be fully removed. All other -Xlp options, as were covered in this design will be deprecated.

Mainly this is just includes where the option is parsed, as all it's doing is overwriting the current gc metadata page size.
https://github.com/eclipse/openj9/blob/491cde3d0b8383a30689fba244f547d6b9315fce/runtime/gc_modron_startup/mmparse.cpp#L815

Removing the capability of the user specifying the meta data page size, means that the metadata page size will always be set to use the system default page size of the system.
https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L129-L131

This validation would not be necessary.
https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L153-L156

Testing Implementation

CodeCache Testing is implemented here: XlpCodeCacheOptionsTestRunner.java

Objectheap Testing is implemented here: XlpOptionsTest.java

These CodeCache/ObjectHeap Tests use -verbose:sizes output to determine what page size was used to verify each large page size test case. As discussed, -verbose:sizes will need to be changed from displaying -Xlp attributes, to attributes that closely model the new -XX proposed options.

Particularity it's purposed that the verbose output be changed from

  -Xlp:objectheap:pagesize=<Page Size Used>,<Page Type Used on Z/OS>     large page size
                  available large page sizes:
                  < List of pages avaialble on obejct heap >
  -Xlp:codecache:pagesize=<Page Size Used>,<Page Type Used on Z/OS>  large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  < List of pages avaialble on code cache>

To:

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: <Page Size Used By Code Cache>
   Page Type Used: <Page Type used on Z/OS>
   Available Page Sizes on Code Cache:
   <List of avaialble page sizes on the code cache>
-XX:[+/-]UseLargePagesObjectHeap
   Page Size Used: <Page Size Used by Object Heap>
   Page Type Used: <Page type used by Object Heap on Z/OS>
   Available Page Sizes on Object Heap:
   < List of avaialble page sizes on the object heap>

As each the codecache, and object heap test expects the output to be in the former -Xlp form, this needs to be adapted to conform with the verbose layout changes.
The test will also need to be adapted to recongize expected warnings when warnings are enabled.

Summary of Changes

Doc Changes
| Description | PR |
| -------------| -------------|
| Add Docs for new -XX Options | TBD |
| Changes to -Xlp, -Xlp:codecache, and -Xlp:objectheap | TBD |

Test Changes
| Description | PR |
| ------------- | ------------- |
| Add Tests for new -XX options | TBD |
| Add j9vmemtests to verify omrvmem_find_valid_page_size & omrvmem_default_large_page_size_ex with OMRPORT_VMEM_PAGE_FLAG_PREFER_PAGEABLE | https://github.com/eclipse/omr/pull/4918 https://github.com/eclipse/openj9/pull/8791 |

Related Changes
| Description | PR |
| ------------- | ------------- |
| Enable XlpOptions, and XlpCodeCache tests | https://github.com/eclipse/openj9/pull/8576 |
| Expand XlpCodeCache tests to include variations | https://github.com/eclipse/openj9/pull/8025 |

References and Footnotes

[1] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1465-L1694
[2] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L587-L670
[3] https://github.com/eclipse/openj9/blob/6a33c82664bdc5c0ac8860c85ab8fdcf18bee1ec/runtime/vm/jvminit.c#L1614
[4] https://github.com/eclipse/openj9/blob/491cde3d0b8383a30689fba244f547d6b9315fce/runtime/oti/j9nonbuilder.h#L4780
[5] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1458-L1461
[6] https://github.com/eclipse/openj9/blob/f7c40c6c6ac97c5bdad13336ab2360f3efb9ff8b/runtime/compiler/control/J9Options.cpp#L1696-L1784
[7] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L583-L584
[8] https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L674-L719
[9] https://github.com/eclipse/openj9/blob/8e50c296bba256a5280e4529a6995b3e5e200b9e/runtime/gc_modron_startup/mmparse.cpp#L356-L357
[10] Consumed Arguments: Unique command line arguments must be marked as consumed for control purposes. Duplicate arguments that are later re-defined are automatically consumed. Each option will have at most one unconsumed argument that needs to be consumed when that argument is processed.

[AlenBadel]

@dmitripivkine Would you have any remaining concerns?

@joransiu I would appreciate if you could give a quick approval on the Z/OS related content.

[joransiu]

The z/OS proposal for handling -XX:+UseLargePages and -XX:LargePageSizeInBytes= looks good to me. Thanks!

[dmitripivkine]

As top level description item it looks reasonable for me.

[AlenBadel]

@pshipton @dmitripivkine
The official docs for -Xlp does not specify precedence rules. Within the objectheap implementation, and tests we assume that if -Xlp is passed with any other arrangement of -Xlp* options it is always ignored. This happens even if it is the right-most option. The only time it is a valid option is if it is passed without any other -Xlp* option.

Was this intended? If so then it needs to be documented since it doesn't follow logical precedence.

E.g -Xlp32M -Xlp

https://github.com/eclipse/openj9/blob/c560cd75ea7ca9c9f2442d2eab7b1afa8a15af1e/runtime/gc_modron_startup/mmparse.cpp#L587-L590

https://github.com/eclipse/openj9/blob/3dab54528080af0731e11858403a37ed962728d8/test/functional/VM_Test/src/j9vm/test/xlp/XlpOptionsTestRunner.java#L172-L177

[DanHeidinga]

I wrote up a proposal on how to handle these options looking at what we'd do today if they were completely new options being added. We obviously don't have a blank slate so we need to figure out how to handle both the new and old options.

I've spent a lot of time thinking about this since you slacked me about it and I had an insight over the weekend. We'd never add the -Xlp:... option forms today in that format. They would be split into individual -XX:[+-] options for easier parsing and clearer intent. Allowing each component to examine the ones specific to it.

That leads to the key insight - the old options should be mapped to the new form rather than trying to make the new fit the old.

See https://github.com/eclipse/openj9/pull/7476#issuecomment-637601836 for my description of how we should do this.

For you question above:

Was this intended? If so then it needs to be documented since it doesn't follow logical precedence.

I think this falls out automatically with the new approach:
-Xlp -> -XX:+UseLargePages
-Xlp4k -> -XX:LargePageSizeInBytes=4k

and both of those are valid regardless of the order they occur in.

[AlenBadel]

One possible change of legacy I see happening is related to a decision we've made in the last design of -XX:+UseLargePages on z/OS. Using -Xlp will always request 1M fixed large pages instead of pageable. We've made the decision that pageable large pages being more performant and should be used within -XX:+UseLargePages. We have a number of users that depend on the legacy behaviour of -Xlp since there are a number of functional trade-offs between fixed and pageable on z/OS.

For now, I suggest we have a special version of -Xlp that is not internally mapped to -XX:+UseLargePagesObjectHeap until EOL of Java 8 where the option is set to be deprecated.

All other Xlp* variants can easily be mapped.

[AlenBadel]

From your description it looks like we need the following options.

-XX:[+/-]UseLargePages
-XX:[+/-]UseLargePagesObjectHeap
-XX:[+/-]UseLargePagesCodeCache

-XX:LargePageSizeInBytes=<Size>
-XX:CodeCacheLargePageSizeInBytes=<Size>
-XX:ObjectHeapLargePageSizeInBytes=<Size>

I suspect we're missing an option, or sub-option that allows us to specify [non]pageable on z/OS.
Looking back at -Xlp, if we had an option to toggle page type we could easily map -Xlp to:
-XX:+UseLargePagesObjectHeap -XX:LargePageType=nonpageable.
I'm not sure how practical a z/OS only option sounds.

[DanHeidinga]

I'd prefer to avoid suboptions when possible, so -XX:LargePageType=nonpageable would be -XX:[+-]UseNonpageableLargePages.

And that seems like a reasonable way to handle -Xlp on zOS

[DanHeidinga]

Are all three sizes needed? Do we have any evidence of users specifying different large page sizes for the object heap vs code cache?

[AlenBadel]

Are all three sizes needed? Do we have any evidence of users specifying different large page sizes for the object heap vs code cache?

I'm not sure. @pshipton Could you chime in?
From the research I've done there doesn't seem to be such a thing on Hotspot - perhaps for good reason?

[pshipton]

I don't know what options are used by customers. I suspect some customers set large pages for one or the other but not both, but not sure how I know this.

We already support options to set the objectheap and codecache sizes separately. We could get away with not adding -XX:LargePageSizeInBytes=<Size> since this is just a shortcut for setting both the other options, except its needed for compatibility.

[DanHeidinga]

The options should either start with the 'ObjectHeap|CodeCache` or end with that. Current proposal changes form:

-XX:[+/-]UseLargePages
-XX:[+/-]UseLargePagesObjectHeap
-XX:[+/-]UseLargePagesCodeCache
-XX:LargePageSizeInBytes=<Size>
-XX:CodeCacheLargePageSizeInBytes=<Size>
-XX:ObjectHeapLargePageSizeInBytes=<Size>

Better to be consistent and always prepend or append it so:

-XX:[+/-]UseLargePages
-XX:[+/-]ObjectHeapUseLargePages
-XX:[+/-]CodeCacheUseLargePages
-XX:LargePageSizeInBytes=<Size>
-XX:CodeCacheLargePageSizeInBytes=<Size>
-XX:ObjectHeapLargePageSizeInBytes=<Size>

or

-XX:[+/-]UseLargePages
-XX:[+/-]UseLargePagesObjectHeap
-XX:[+/-]UseLargePagesCodeCache
-XX:LargePageSizeInBytes=<Size>
-XX:LargePageObjectHeapSizeInBytes=<Size>
-XX:LargePageCodeCacheSizeInBytes=<Size>

[AlenBadel]

I've went ahead and posted the design doc of the current paging options, as well as the proposed additions we've discussed.

See https://github.com/eclipse/openj9/issues/8671#issue-571666508

@dmitripivkine @DanHeidinga @pshipton Feel free to review the behaviour and let me know if anything is unclear or inaccurate.

[pshipton]

lgtm

[AlenBadel]

@pshipton Is it right to assume that the irregularity described in https://github.com/eclipse/openj9/issues/8671#issuecomment-637582166 is a bug and that the option precedence for -Xlpshould follow all other -Xlp* options?

[DanHeidinga]

Thanks for pulling this together @AlenBadel.

A couple of comments:

• Deprecation after 8
  
  
  
  • I would prefer we deprecate in 8 so it's very clear that users should adopt the new -XX: options instead. Deprecating doesn't mean we remove them immediately.
  
  
• Row 5 (-Xlp) has two different mappings depending on whether the OS is zOS or not.
  
  
  
  • I would prefer to keep the base mapping the same on all platforms and treat os specific bits as additional flags. This would lead to:
  
  

Old Option | Mapping
------------ | -------------
-Xlp | -XX:+UseLargePagesObjectHeap
-Xlp | (z/OS) -XX:+UseLargePagesObjectHeap -XX:+UseNonpageableLargePages

• Row 6 (Xlp<size>) indicates that -XX:+UseLargePages is a short form for -XX:+UseLargePagesObjectHeap -XX:+UseLargePagesCodeCache. Which is fine, just calling it out for everyone's awareness.

• Row 12 adds a new -XX:+UseNonpageableLargePages option and indicates it only applies to the Object heap.

  • If it's only object heap, it should be renamed to be clear on that fact: -XX:+UseNonpageableLargePagesObjectHeap

For the question about Precedence rules, we don't need to worry about "last applicable option wins". The + & - forms can be applied left to right and the final configuration read. Same for the sizes. This is partly why I prefer the -XX:[+-] forms.

[AlenBadel]

Thanks for the review and clarification @DanHeidinga. I've made the changes.

[pshipton]

Is it right to assume that the irregularity described in https://github.com/eclipse/openj9/issues/8671#issuecomment-637582166 is a bug

Certainly that makes things much easier than trying to replicate the current behavior once we start mapping -Xlp options to -XX: options. However it could mean a change in behavior for a customer when updating, so we should make sure we document this in the release notes.

[dmitripivkine]

Is it right to assume that the irregularity described in #8671 (comment) is a bug

Certainly that makes things much easier than trying to replicate the current behavior once we start mapping -Xlp options to -XX: options. However it could mean a change in behavior for a customer when updating, so we should make sure we document this in the release notes.

I believe this is intentional. -Xlp supposed to be ignored if -Xlp<size> is provided. It is like "I want large pages and size of large page should be <size>". I believe this is kind of "translation" of -XX:+UseLargePages -XX:LargePageSizeInBytes=<Size>

[gita-omr]

What should happen if -XX:LargePageSizeInBytes=<Size> -XX:+UseLargePages is specified?

[AlenBadel]

What should happen if -XX:LargePageSizeInBytes=<Size> -XX:+UseLargePages is specified?

Logically, -XX:LargePageSizeInBytes=<Size> should be ignored. It doesn't make sense that an argument be passed before the feature is enabled. I assume there's a global OpenJ9 option convention rule to address this.

I believe this is intentional. -Xlp supposed to be ignored if -Xlp is provided

If was intentional, it wasn't documented. Let's agree to treat all options equally, and not have -Xlp overwritten. As Peter mentioned, it should a reasonable action as long as it's properly documented and we've marked these options for deprecation.

[dmitripivkine]

I believe different options should be context independent (order irrelevant). My point was in this context -Xlp (enable LP) and -Xlp<size> (specify size for LP) should be treated at two different options (not a two forms of the same option). if somebody specifies (on AIX) -Xlp64k -Xlp LP size selected should be 64k, not 16m default and order does not matter: -Xlp -Xlp64k should be equivalent.

[pshipton]

I agree with Dmitri.
-XX:LargePageSizeInBytes=<Size> -XX:+UseLargePages should use <Size>, not ignore the size because it comes first.

[AlenBadel]

Agreed, options will be proceed no matter of the order, unless they are overwritten, or disabled by an option at a later index.

I certainly misspoke. We wouldn't need any special cases or precedence for -Xlp. Ultimately it will be mapped to -XX:+UseLargePages, and any option bearing a size argument (-Xlp<size>, -Xlp:[codecache/objectheap]) will effectively dictate the page size used. All legacy is preserved without needing any special cases.

[AlenBadel]

I've updated the doc to reflect the prior discussion. Namely the changes are:

• Xlp options will be deprecated in Java 8, and set to be removed when Java 8 is EOL.
• Renamed -XX:+UseNonpageableLargePages to XX:+UseNonpageableLargePagesObjectHeap
• Updated Precedence rules to reflect how -Xlp will be used with these new -XX options.

[AlenBadel]

@DanHeidinga & @dmitripivkine Would you have any further concerns?

@joransiu, @zl-wang @andrewcraik Just as an FYI about the deprecation of the -Xlp options. The OpenJ9 docs outlined that we would deprecate only -Xlp, and -Xlp<size> after Java 8; keeping -Xlp:[codecache/objectheap] options. Here, we're proposing to deprecate all -Xlp* options starting on Java8, and users will ultimately see a warning when using these options.

[andrewcraik]

What is the plan with regard to still being able to tune the codecache and objectheap sizes independently for performance investigations etc? Are there equivalent options in the new scheme?

[AlenBadel]

What is the plan with regard to still being able to tune the codecache and objectheap sizes independently for performance investigations etc? Are there equivalent options in the new scheme?

I've linked a spreadsheet which provides explanations of the current -Xlp option behaviour, the new -XX options proposed, and how the older options will be mapped to the new proposed options internally. See: https://github.com/eclipse/openj9/issues/8671#issue-571666508

The proposed addition to of -XX:LargePageSizeInBytesCodeCache=<size>, and -XX:LargePageSizeInBytesObjectHeap=<size> will allow users to independently set the codecache, and objectheap sizes respectively. Keeping in mind that large pages will need to be enabled as well with (-XX:+UseLargePages[Codecache/ObjectHeap]) before it would take effect. Hence why something like -Xlp:codecache:pagesize=<size> is mapped as -XX:+UseLargePagesCodeCache -XX:LargePageSizeInBytesCodeCache=<size>.

[dmitripivkine]

Keeping in mind that large pages will need to be enabled as well with (-XX:+UseLargePages[Codecache/ObjectHeap]) before it would take effect.

I am not sure this is correct behaviour. Why if for example -XX:LargePageSizeInBytesObjectHeap=<size> is requested it requires to use another option -XX:+UseLargePagesObjectHeap to be "enabled"? It is self-complete option. I believe it should take effect regardless

[AlenBadel]

From my understanding this was an OpenJ9 option convention.

As @DanHeidinga mentioned in https://github.com/eclipse/openj9/pull/7476#issuecomment-637601836

The current -Xlp:... options would look very different if they were being developed today. Each sub-option would be its own -XX:[+-] form. So -Xlp:objectheap:pagesize=2G would be written as -XX:+UseLargePagesObjectHeap -XX:LargePageSizeInBytes=2G

[AlenBadel]

FYI @mstoodle @0xdaryl. Feel free to chime in on this proposal.

[DanHeidinga]

Why if for example -XX:LargePageSizeInBytesObjectHeap= is requested it requires to use another option -XX:+UseLargePagesObjectHeap to be "enabled"? It is self-complete option. I believe it should take effect regardles

We're mapping to Hotspot's behaviour, which as far as I can tell from -XX:+PrintFlagsFinal, doesn't enable large pages unless explicitly requested.

Our legacy size options become the equivalent of an enable + the setting the size.

[andrewcraik]

Keeping in mind that large pages will need to be enabled as well with (-XX:+UseLargePages[Codecache/ObjectHeap]) before it would take effect.

I am not sure this is correct behaviour. Why if for example -XX:LargePageSizeInBytesObjectHeap= is requested it requires to use another option -XX:+UseLargePagesObjectHeap to be "enabled"? It is self-complete option. I believe it should take effect regardless

If we do require the enablement option to make the page size setting take effect then I would hope we could generate a warning to help people avoid falling into the trap of setting the size but not setting the enablement...

[dmitripivkine]

Why if for example -XX:LargePageSizeInBytesObjectHeap= is requested it requires to use another option -XX:+UseLargePagesObjectHeap to be "enabled"? It is self-complete option. I believe it should take effect regardles

We're mapping to Hotspot's behaviour, which as far as I can tell from -XX:+PrintFlagsFinal, doesn't enable large pages unless explicitly requested.

Our legacy size options become the equivalent of an enable + the setting the size.

Ok, I was under impression that Hotspot's option to set size enable feature implicitly

[mstoodle]

I appreciate all the attention being brought to this issue to ensure we build the right solution.

Since I was asked (and not because I think it really adds anything to this conversation), I support the direction @DanHeidinga has been recommending in this issue. I agree with trying to process options in a way consistent with how Hotspot processes its options. Policies like "setting the large page size also enables large pages" makes for complicated rules to reason about for both the JVM and for people (e.g. what's the rightmost large page enabling option versus the rightmost -XX:-UseLargePages). It's also hard to be complete and future proof.

Note that having full -XX:+PrintFlagsFinal support would help with (though not solve) the issue @andrewcraik raised. Warning messages seem helpful but can also unintentionally cause pain.

[andrewcraik]

I'm fine with maximum compatibility. A warning would be nice but I do see the point @mstoodle is making. If there is an option to warn about ignored parameters (which I think we already have) then that would satisfy me since I can just make sure to always run with that to validate any configurations I build with options.

[pshipton]

https://www.eclipse.org/openj9/docs/xxignoreunrecognizedxxcolonoptions/

[AlenBadel]

Great, if -XX:LargePageSizeInBytes, or any of it's variants are specified but the respective component's large page functionality has not been enabled, and -XX:-IgnoreUnrecognizedXXColonOptions is used we'll simply provide an error.

[pshipton]

and -XX:-IgnoreUnrecognizedXXColonOptions is used we'll simply provide an error.

No, they are recognized options. They should not cause any error, but be consumed without having any effect.

[pshipton]

To have warnings about invalid large page configurations we should add another option like -XX:+LargePageWarnings. Note the existing -Xlp:objectheap option supports [,strict|warn] options, and these should also have -XX: equivalents before deprecating -Xlp:objectheap.
https://www.eclipse.org/openj9/docs/xlpobjectheap/

[AlenBadel]

To support strict, and warn we would need an additional two options. -XX:[+/-]LargePageWarnings(warn), and -XX:[+/-]LargePageErrors(strict) keeping precedence rules as If both strict and warn are specified, strict takes precedence.

Since these options will only impact the objectheap, would we want to name them -XX:[+/-]LargePageWarningsObjectHeap, and -XX:[+/-]LargePageErrorsObjectHeap similarly to https://github.com/eclipse/openj9/issues/8671#issuecomment-639693280?

Row 12 adds a new -XX:+UseNonpageableLargePages option and indicates it only applies to the Object heap.
    If it's only object heap, it should be renamed to be clear on that fact: -XX:+UseNonpageableLargePagesObjectHeap

[DanHeidinga]

Since these options will only impact the objectheap, would we want to name them -XX:[+/-]LargePageWarningsObjectHeap, and -XX:[+/-]LargePageErrorsObjectHeap similarly to #8671 (comment)?

Yes.

[AlenBadel]

Great. I've updated the design to include:

• Descriptions for -XX:[+/-]LargePageWarningsObjectHeap, and -XX:[+/-]LargePageErrorsObjectHeap.
• How the existing -Xlp:objectheap:pagesize=<size>,[warn/strict] will be mapped to these options.
• Precedence Rules for both these options.

[AlenBadel]

-XX:[+/-]LargePageWarnings, and -XX:[+/-]LargePageErrors are better naming options because we ultimately need options that enable warnings, and errors for both the codecache and objectheap.

-XX:[+/-]LargePageWarnings

(ObjectHeap)
Causes a warning message to be generated if large pages are requested but cannot be obtained. Also, this option will also cause a warning message if the user has set a large apge size (-XX:LargePageSizeInBytes, -XX:LargePageSizeInBytesObjectHeap), but has not enabled Large Pages (-XX:+UseLargePages, or -XX:+UseLargePagesObjectHeap). This option allows the VM to continue.

(CodeCache)
Causes a warning message to be generated if the user has set a large page size(-XX:LargePageSizeInBytes, or -XX:LargePAgeSizeInBytesCodeCache), but large pages are not enabled (-XX:+UseLargePages, or -XX:+UseLargePagesCodeCache). This option allows the VM to continue.

-XX:[+/-]LargePageErrors
(ObjectHeap)
Causes an error message to be generated if large pages are requested but cannot be obtained. Also causes an error message if the user has set a large page size (-XX:LargePageSizeInBytes, -XX:LargePageSizeInBytesObjectHeap), but has not enabled Large Pages (-XX:+UseLargePages, or -XX:+UseLargePagesObjectHeap). This option causes the VM to end.

(CodeCache)
Causes an error message to be generated if the user has set a large page size(-XX:LargePageSizeInBytes, or -XX:LargePAgeSizeInBytesCodeCache), but large pages are not enabled (-XX:+UseLargePages, or -XX:+UseLargePagesCodeCache). This option allows the VM to end.

An issue I see occurs when mapping these options to -XX:objectheap:pagesize=<size>,[strict/warn]. If the strict/warn sub-option is mapped to LargePageErrors, and LargePageWarnings respectively this will enable errors and warnings not only for the objectheap, but also the codecache. I don't see how enabling warnings for the codecache would be imperative for the user but we would be introducing additional warning, and error path that wasn't part of the legacy xlp behaviour. The question is if this is acceptable, or do we need objectheap variant options for warnings and errors.

[gita-omr]

An issue I see occurs when mapping these options to -XX:objectheap:pagesize=<size>,[strict/warn]. If the strict/warn sub-option is mapped to LargePageErrors, and LargePageWarnings respectively this will enable errors and warnings not only for the objectheap, but also the codecache. I don't see how enabling warnings for the codecache would be imperative for the user but we would be introducing additional warning, and error path that wasn't part of the legacy xlp behaviour. The question is if this is acceptable, or do we need objectheap variant options for warnings and errors.

It's a good point. But mapping is really an implementation issue, is not it? We don't have to map, just set a flag "warn about objectheap" when we see -XX:objectheap:pagesize=<size>,warnand set "warn about objectheap" flag as well as "warn about codecache" when we see XX:+LargePageWarnings

[0xdaryl]

Can someone explain the reason why there isn't currently a strict or warning option for large code pages?

[pshipton]

Unless somebody else has an explanation, I'd say nobody asked for it.

[0xdaryl]

If that's the only reason then for symmetry I suggest that support be added going forward.

[AlenBadel]

It's a good point. But mapping is really an implementation issue, is not it? We don't have to map, just set a flag "warn about objectheap" when we see -XX:objectheap:pagesize=<size>,warnand set "warn about objectheap" flag as well as "warn about codecache" when we see XX:+LargePageWarnings

We've been able to exactly map each -Xlp option, and sub-option to a newly created -XX: option. It would be great to continue doing so without adding a specialty parsing rule. This would prevent us from needing special parsing rules when someone decides to use -Xlp, and large page -XX options together.

If that's the only reason then for symmetry I suggest that support be added going forward.

Extending the question I asked earlier, would it sound clear to everyone that enabling [strict/warn] on -Xlp:codecache, would enable the same on the object-heap without the option being specified and vice versa. Otherwise I can always create three different variations for each warn, and error options. (Objectheap, CodeCache, and Both)

[dmitripivkine]

Unless somebody else has an explanation, I'd say nobody asked for it.

As I remember strict/warning sub options were added as a response to controversial requests:
Some people uses the same java command line across multi platform environment so option value might be not applicable sometimes but they want execute it regardless and some people wants immediate stop. For details see RTC Jazz 93937

[dmitripivkine]

In general I believe there is no need to move existing complexity (sometimes bogus and confusing) to new options world. We should try to keep new options set to be clean and transparent

[AlenBadel]

@dmitripivkine The original request to enable warnings with new options. https://github.com/eclipse/openj9/issues/8671#issuecomment-642314807

Adding of XX:[+/-]LargePageWarnings is a requirement. Are you suggesting we keep this as the only publicly documented warnings option? While we parse the -Xlp strict/warn without directly mapping it to any new option.

[DanHeidinga]

We need both the Warning and the Error forms as they are different sets of behaviour - one is identify an LP problem and one is fail to start if an LP problem occurs.

The proposal is to make those apply equally to codecache and object heap rather than adding variants for both. This gives us fewer new options and leaves room to do something different in the future if required.

[dmitripivkine]

8671 (comment)

I believe XX:[+/-]LargePageWarnings controls output only (print warning or not) and has no impact to JVM logic (continue or terminate). This is easy to understand clean behaviour.
If we need to support JVM termination on incompatible settings (strict equivalent) we should use another option to do it (like XX:[+/-]LargePageErrors suggested by you).

[AlenBadel]

We need both the Warning and the Error forms as they are different sets of behaviour - one is identify an LP problem and one is fail to start if an LP problem occurs.

The proposal is to make those apply equally to codecache and object heap rather than adding variants for both. This gives us fewer new options and leaves room to do something different in the future if required.

Agreed. To decouple the complexity:

• -XX:[+/-]LargePageWarnings Enable/Disable LP Warnings on CodeCache and ObjectHeap
• -XX:[+/-]LargePageErrors Enable/Disable LP Errors on CodeCache and ObjectHeap
• -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn]. Will be parsed individually, and not mapped to any -XX option.

Precedence Rules: Errors will always take precedence over Warnings if both are enabled.

[DanHeidinga]

• -XX:[+/-]LargePageWarnings Enable/Disable LP Warnings on CodeCache and ObjectHeap

Yes

• -XX:[+/-]LargePageErrors Enable/Disable LP Errors on CodeCache and ObjectHeap

Yes

• -Xlp:[codecache/objectheap]:pagesize=<size>,[strict/warn]. Will be parsed individually, and not mapped to any -XX option.

No. These should be mapped and warnings / errors apply to both CodeCache and ObjectHeap. (And yes that's a change in behaviour but it's a pretty unlikely to occur case)

[AlenBadel]

Great. Updated the design to reflect this. I think we're on all the same page - no pun intended.

[AlenBadel]

I welcome anyone with questions or suggestions to come forth. I'm willing to provide further clarifications if anyone has outstanding questions or concerns before moving forward.

[AlenBadel]

In addition to -XX:[+|-]UseNonpageableLargePagesObjectHeap, I think we should add -XX:[+|-]UsePageableLargePagesObjectHeap.

This option would help anyone that is explicitly trying to use pageable large pages. Using this option with -XX:+LargePage[Warnings/Errors] would help users manage and control page type usage on Z/OS.

[AlenBadel]

In order to fully remove parsing within the GC, we will need to address the undocumented option -Xlp:gcmetadata:pagesize=<size>.

This option will be deprecated along with all other -Xlp options. Would there be any interest in creating equivalent -XX options for this and properly documenting these options?

[dmitripivkine]

In order to fully remove parsing within the GC, we will need to address the undocumented option -Xlp:gcmetadata:pagesize=<size>.

This option will be deprecated along with all other -Xlp options. Would there be any interest in creating equivalent -XX options for this and properly documenting these options?

I believe because this option is not documented we don't need deprecate it, even we don't need to keep it. Instead (if we want to keep this functionality) we should provide an alternative way to setup it using new -XX command. So in case if somebody still use it the new -XX option should be used instead

[dmitripivkine]

In addition to -XX:[+|-]UseNonpageableLargePagesObjectHeap, I think we should add -XX:[+|-]UsePageableLargePagesObjectHeap.

I believe this would be very confusing. There are only two types of pages: pageable and non-pageable. So in your suggestion -XX:+UseNonpageableLargePagesObjectHeap means -XX:-UsePageableLargePagesObjectHeap and vice versa. I believe we need only one of them. I would prefer -XX:[+|-]UsePageableLargePagesObjectHeap

[pshipton]

-XX:[+|-]UsePageableLargePagesObjectHeap works for me.

Not that I'm suggesting it's better, but fyi another option is -XX:zOSLargePagesObjectHeap=<pageable|nonpageable>.

[AlenBadel]

I believe this would be very confusing. There are only two types of pages: pageable and non-pageable. So in your suggestion -XX:+UseNonpageableLargePagesObjectHeap means -XX:-UsePageableLargePagesObjectHeap and vice versa. I believe we need only one of them. I would prefer -XX:[+|-]UsePageableLargePagesObjectHeap

Let's consider the following arguments are used on Z/OS:

Scenario A:
-XX:+UseLargePages -XX:LargePageInBytes=1M -XX:+UseNonpageableLargePagesObjectHeap -XX:+LargePageWarnings

Configured Pages: [1M Pageable]
A Warning would be shown saying that 1M fixed large pages are not configured, and the ObjectHeap would be using 1M pageable instead.

Scenario B:
-XX:+UseLargePages -XX:LargePageInBytes=1M -XX:+LargePageWarnings

Configured Pages: [1M Fixed]
A warning would not show, because the default behaviour is to seek 1M pageable preferably but use 1M fixed if pageable is not available. See: https://github.com/eclipse/omr/pull/4762 PREFER_PAGEABLE.

The reasoning here is that there's no way for the user to explicitly specify they're looking to use pageable large pages if that's the only type they're looking to use. The default behaviour without specifying a type is to just use pageable preferably with a fall back to the fixed; which is fine if the user doesn't care what type is used.

[dmitripivkine]

The problem I see we should not have two options for the same thing (just inverse). It creates dependability between options (controversial request needs to be controlled). I believe -XX options should be context free as much as possible (independent) except cases when -XX option plays role of sub option (like -XX:+UseLargePages -XX:LargePageInBytes=1M)

[dmitripivkine]

-XX:[+|-]UsePageableLargePagesObjectHeap works for me.

Not that I'm suggesting it's better, but fyi another option is -XX:zOSLargePagesObjectHeap=<pageable|nonpageable>.

Any of this two is ok for me (just tried to avoid "not non-pageable")

[AlenBadel]

-XX:zOSLargePagesObjectHeap=<pageable|nonpageable>

Agreed. This would definitely solve our problem. As long as everyone stays in agreement I'll make that change.

[AlenBadel]

@DanHeidinga @pshipton
What's the best way to proceed with -Xlp:gcmetadata?

We've established that it's an undocumented option and is not used by any OpenJ9 test.

Should the option be either removed totally, deprecated, or re-implemented using -XX convention? It's worth nothing that -Xlp:gcmetadata currently supports every sub-option that -Xlp:objectheap supports. This means that we would need multiple -XX options if we choose to re-implement the option.

See
https://github.com/eclipse/openj9/issues/8671#issuecomment-649802595

[pshipton]

If @dmitripivkine wants to keep the option, we'll need -XX: options for it.

-XX:[+|-]UseLargePagesGCMetadata
-XX:LargePageSizeInBytesGCMetadata=<size>
-XX:zOSLargePagesGCMetadata=<pageable|nonpageable>

I expect -XX:+LargePageWarnings, -XX:+LargePageErrors report for GCMetadata as well.

[DanHeidinga]

My preference would be to remove it during this transition unless @dmitripivkine objects. Don't introduce a new form for it and stop recognizing the -Xlp form of the gcmetadata option

[dmitripivkine]

My preference would be to remove it during this transition unless @dmitripivkine objects. Don't introduce a new form for it and stop recognizing the -Xlp form of the gcmetadata option

I agree. I believe there is no use case when allocation of GC Metadata in LP gains measurable performance difference.
@amicic Do you agree?

[amicic]

+1

[AlenBadel]

I've went ahead and updated an implementation overview. https://github.com/eclipse/openj9/issues/8671#issue-571666508

@dmitripivkine @pshipton @gita-omr
I appreciate any feedback. Particularly, I've specified two approaches to implement central parsing.
(1) Iterating though vm arguments left to right and using pattern matching.
(2) Using FIND_AND_CONSUME_ARG

Any opinion on which one we should proceed with would be appreciated.

[gita-omr]

I definitely like (1) more. It just seems more intuitive and easier to maintain. Note that (2) not just uses FIND_AND_CONSUME_ARG but needs to group options in some specific groups, find indexes for all options in the group, then carefully compare those indexes to figure out the precedence. Not sure what happens when an option belongs to more than one group, or new option is added.

[dmitripivkine]

Case (1) has hardcoded preference "+" over "-" option.
Also using of macro adding an abstract layer for extra functionality potentially: for example hidden (consumed) option handling, mapped option etc.

There is an example of handling -XX +/- option from current code:

            /* -XX commandline option for +/- TransparentHugepage */
            argIndex = FIND_ARG_IN_VMARGS(EXACT_MATCH, VMOPT_XXNOTRANSPARENT_HUGEPAGE, NULL);
            argIndex2 = FIND_ARG_IN_VMARGS(EXACT_MATCH, VMOPT_XXTRANSPARENT_HUGEPAGE, NULL);
            {
                /* Last instance of +/- TransparentHugepage found on the command line wins
                 *
                 * Default to use OMR setting (Enable for all Linux with THP set to madvise)
                 */
                if (argIndex2 > argIndex) {
                    j9port_control(J9PORT_CTLDATA_VMEM_ADVISE_HUGEPAGE, 1);
                } else if (argIndex > argIndex2) {
                    j9port_control(J9PORT_CTLDATA_VMEM_ADVISE_HUGEPAGE, 0);
                }
            }

An essential in this example:

• always handle "+" and "-" of the option as a pair
• most right wins (if both provided)
• I believe using parsing macro is preferable over strcmp

I don't see why not to use the same pattern for new options?

[AlenBadel]

Not sure what happens when an option belongs to more than one group, or new option is added.

As large pages is concerned I cant fathom an option being added that impacts more than one of these functional groups. It's exactly why we added multiple new options. We've designed this so that we are creating building blocks that can build any possible large page scenario. I don't see the need to map a specific profile, or configuration to a single -XX option. That's the main motivation behind moving away from -Xlp.

[gita-omr]

I think handling +/- can be fixed, as well as using macros vs strcmp. I think the main question is: do we want to just have one pass over all the command line arguments and keep setting flags accordingly or we want to create groups of options, look for those options and then compare their indexes? Maybe the latter is the way it's been done historically but I find it a bit complicated.

[dmitripivkine]

I think handling +/- can be fixed, as well as using macros vs strcmp. I think the main question is: do we want to just have one pass over all the command line arguments and keep setting flags accordingly or we want to create groups of options, look for those options and then compare their indexes? Maybe the latter is the way it's been done historically but I find it a bit complicated.

My vote is to keep handling of new -XX options simple. As far as each new option is independent (but might have relation option/sub-option like -XX:+UseLargePages -XX:LargePageInBytes=1M) the handling should be very simple in a new world. There are two areas of application ("object heap" and "code cache") and three groups of options (general LP request, sub-options to specify size/type of page and caution level). The grammar is really simple - and should be significantly less complicated.

• get winner inside each option pair if any
• for object heap get general enabled/disabled, get page size/type specified or default
• for code cache get general enabled/disabled, get page size specified or default
• get caution level

I think we should rename -XX:zOSLargePagesObjectHeap=<type> to -XX:zOSLargePagesObjectHeapType=<type> for clarity. Does it make sense?

There is one more aspect:
Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

[AlenBadel]

I think we should rename -XX:zOSLargePagesObjectHeap=<type> to -XX:zOSLargePagesObjectHeapType=<type> for clarity. Does it make sense?

I agree, that makes sense. Will make the change to the design now.

Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

This is true not only for AIX, but for other platforms. Historically, when -Xlp is not specified the codecache will always try to allocate a preferred page size for each platform.

https://github.com/eclipse/openj9/blob/270a41478fa4f73089a001892c021c47ef39a5a6/runtime/compiler/control/J9Options.cpp#L1748-L1758

We can resolve this by adding a new option. Something like -XX:[+/-]UsePreferredPages. Mapping the current behaviour to this option is always enabled by default. When the option is disabled, we could instead just use the smallest OS configured page size which is accessilbe via index 0 of j9vmem_supported_page_sizes.

[pshipton]

I'll put my vote in for option 2. It think it's easier to understand, even though there is some extra work to track the indexes in order to support -Xlp. Down the road once -Xlp parsing is removed, we'll be left with a cleaner solution that is similar to all the other parsing.

[gita-omr]

Ok, since we have 3:1 vote for approach (2) I think we can continue implementing it. Hopefully @DanHeidinga agrees with it as well.

[AlenBadel]

I've went ahead and made the following changes to the design/implementation:

• Rename -XX:zOSLargePagesObjectHeap=<type> to -XX:zOSLargePagesObjectHeapType=<type>.
• Removed Option 1, and updated the remainder of the documentation to reflect variable naming uniform with option 2.
• Added implementation plans and details to add test cases and make modifications to Xlp CodeCache, and Objectheap java tests and -verbose:sizes.

[AlenBadel]

Re-iterating an issue that @dmitripivkine brought up:

Currently If default page size is "Large" (like 64K on AIX) the only way to request allocation in 4K ("Small") pages is to use -Xlp4k or -Xlp:objectheap:size=4k. This is confusing (and even inconsistent does not allowed on ZOS). I believe it would it be sufficient to use 4K pages if LP is explicitly not enabled (obviously). But should we allow requests for 4K page like -XX:+UseLargePages -XX:LargePageInBytes=4k?

Would anyone be interested in this proposed solution https://github.com/eclipse/openj9/issues/8671#issuecomment-662538479? Allowing the user to specify a new option like -XX:-UsePreferredPages rather than specifying -XX:+UseLargePages -XX:LargePageInBytes=4k.

[pshipton]

Just noting it should be XX:-UsePreferredPagesObjectHeap, etc. If -XX:+UseLargePages -XX:LargePageInBytes=4k works, I suggest UsePreferredPages not be included in the initial implementation, but added later if desired.

[dmitripivkine]

In theory to use 4k (small) pages we can specify "do not use LP" like -XX:-UseLargePages and I believe it should be enough

[AlenBadel]

In theory to use 4k (small) pages we can specify "do not use LP" like -XX:-UseLargePages and I believe it should be enough

Don't we need to document a default state for each option? I.e by default JVM will be initialized with-XX:-UseLargePages if it is not explicitly enabled by the user.

[AlenBadel]

Preferred page sizes are actually very similar to sizes used by -XX:+UseLagePages.

On the codecache:

| Platform | -XX:+UseLargePages | PreferredPageSize |
|---------| --------| -----|
| Z/OS | 1MB Pageable | 1MB Pageable |
| AIX/pLinux | 64K/16M | 64K |
| XLinux/Windows | Usually 2M | 2M |

The only difference here is that using -XX:+UseLargePages will allow AIX/pLinux to use 16M pages if configured.
Would there be any reason not to remove the hard coded preferred pages in favour for enabling -XX:+UseLargePages by default? I.e allowing AIX/pLinux to use up to 16M pages by default.

If we're able to remove these preferred page sizes, then we could describe -XX:-UseLargePages to use 4k pages as @dmitripivkine has suggested here. https://github.com/eclipse/openj9/issues/8671#issuecomment-664616505

Would there be any performance implications?

[AlenBadel]

@dmitripivkine @DanHeidinga Would there be any objection to enabling -XX:+UseLargePages by default?

[dmitripivkine]

@dmitripivkine @DanHeidinga Would there be any objection to enabling -XX:+UseLargePages by default?

Why do you think it is necessary? We have wide list of platforms (Linux X,P,Z; Windows, AIX, OSX, ZOS all of them should be considered 64 or 32 bits). This is list of page sizes used for object heap by default:

#if defined(AIXPPC)
    requestedPageSize = SIXTY_FOUR_KB; /* Use 64K pages for AIX-32 and AIX-64 */
#elif ((defined(LINUX) || defined(OSX)) && (defined(J9X86) || defined(J9HAMMER)))
    requestedPageSize = TWO_MB; /* Use 2M pages for Linux/OSX x86-64 */
#elif (defined(LINUX) && defined(S390))
    requestedPageSize = ONE_MB; /* Use 1M pages for zLinux-31 and zLinux-64 */
#elif defined(J9ZOS390)
    requestedPageSize = ONE_MB; /* Use 1M PAGEABLE for ZOS-31 and ZOS-64 */
    requestedPageFlags = OMRPORT_VMEM_PAGE_FLAG_PAGEABLE;
#endif /* defined(AIXPPC) */

As you see not all platforms are assigned to use LPs for heap. And for some of them we don't want to change default. For example for Linux PPC LE 64 bit small page is set to 64k and large page to 1G and I believe it is not good idea to use 1G pages as default.
Another example default LP for AIX is 16m and it will overwrite current default 64k

[AlenBadel]

Linux PPC LE 64 bit small page is set to 64k and large page to 1G

I'm not sure where you're getting the 1G from.
https://github.com/eclipse/omr/blob/796f5d0fe03aa59e10febc044acb98a820f3827b/port/linux/omrvmem.c#L1389-L1426

Strictly speaking on -XX:+UseLargePages, which obtains the default value from the PortLibrary API above.
Objectheap will use the second smallest configured page size. That is usually 64K, since smallest is always 4K.
Codecache will use the system default set by the OS via _SC_PAGESIZE. Usually 64K.

I sense we can make this work by increasing the threshold for non-executable large pages (Object-Heap) to look for 2M or less on linux instead of the 64K it is using now. Which would match the default object heap page size hard-coded in your comment above.

[DanHeidinga]

Would there be any objection to enabling -XX:+UseLargePages by default?

I'd need to see the performance analysis on each platform before we made this change. We've seen regressions in the past when changing the LP behaviour. See previous work around using madvise for transparent large pages.

[AlenBadel]

Great. I'll set -XX:-UseLargePages by default, it will be defined as using the code cache and object heap "preferred pages", or hard coded values above. I will open up another issue after the initial implementation on what to do with preferred pages. To determine if we can either create a new option to disable, and ignore these values, or if we can remove them totally.

[AlenBadel]

I have a concern on how we currently output the selected, and available page sizes using -verbose:sizes.

  -Xlp:objectheap:pagesize=64K   large page size
                  available large page sizes:
                  64K
  -Xlp:codecache:pagesize=64K    large page size for JIT code cache
                  available large page sizes for JIT code cache:
                  64K

Printing out -Xlp:[objectheap/codecache] is misleading when the option is not specified at all.

I propose that we adapt the following structure:

-XX:[+/-]UseLargePagesCodeCache   // State of CodeCache Large Pages
-XX:LagePageSizeInBytesCodeCache=<size> // Will only print codecache large page size requested if enabled.
   <Print Large Page Size Actually Used>
   <Specify if Warnings, or Errors are enabled.> // I.e Warning or Errors. Will not print if none are specified. 
   <Print Available Page Sizes on CodeCache>

-XX:[+/-]UseLargePagesObjectHeap  // State of ObjectHeap Large Pages
-XX:LagePageSizeInBytesObjectHeap=<size> // Will only print objectheap large page size requested if enabled.
-XX:LargePageType=[non/pageable] // Will only print objectheap large page type requested if enabled.
   <Print Large Page Size/Type Actually Used>
   <Specify if Warnings, or Errors are enabled.> // I.e Warning or Errors. Will not print if none are specified. 
   <Print Available Page Sizes on ObjectHeap>

The above changes will mean we will need to change how we parse test results within XlpOptionsTest[1], and XlpCodeCacheTest[2].

@dmitripivkine Do you think this should be changed?

[1] https://github.com/eclipse/openj9/blob/2f295ea91cdd97ff37eb5d164fc63014ea718bc7/test/functional/VM_Test/src/j9vm/test/xlp/XlpOptionsTest.java
[2] https://github.com/eclipse/openj9/blob/2f295ea91cdd97ff37eb5d164fc63014ea718bc7/test/functional/VM_Test/src/j9vm/test/xlpcodecache/XlpCodeCacheOptionsTest.java

[dmitripivkine]

The goal of -verbose:sizes to show chosen / available range of sizes. Whatever we do (even replace some -Xlp commands to any kind of verbose output) sizes must be clear and easy to see.
Would you please provide an example of output you suggesting?

[AlenBadel]

Would you please provide an example of output you suggesting?

-XX:+UseLargePagesCodeCache                    State of Large Pages on CodeCache.
-XX:LagePageSizeInBytesCodeCache=16M   Large Page Size Requested on CodeCache.
   Warnings Enabled
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

-XX:-UseLargePagesCodeCache                    State of Large Pages on CodeCache.
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

The only output that somewhat doesn't belong is if Warnings/Errors are enabled. It has nothing to do with size.

[dmitripivkine]

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

[AlenBadel]

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

I agree. Page data should be anchored under a large page option, or it would look totally misaligned.

I.e

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

-XX:[+/-]UseLargePages[CodeCache/ObjectHeap] should be the only options specified. Page info that pertains to the size selected, type selected, and available combinations will be printed indented under that option.

[dmitripivkine]

My vote would be report actual data only, do not put what is requested. We need output actually selected as well as any other valid choice. Like ' object heap page size selected "A", available "B","C","D" '. I don't think we need to list all set of options can be used as well as how they are treated. This is not a guide for options.

I agree. Page data should be anchored under a large page option, or it would look totally misaligned.

I.e

-XX:[+/-]UseLargePagesCodeCache
   Page Size Used: 64K
   Available Page Sizes on CodeCache:
   64K

-XX:[+/-]UseLargePages[CodeCache/ObjectHeap] should be the only options specified. Page info that pertains to the size selected, type selected, and available combinations will be printed indented under that option.

@pshipton Any suggestions how to reformat output of -verbose:sizes ?

[AlenBadel]

I've updated https://github.com/eclipse/openj9/issues/8671#issue-571666508 to reflex the new proposed format of -verbose:sizes output, and the test changes necessary to conform to the new output.

Mainly, the test currently parses the verbose output as written in the -Xlp fashion and we would need accompanying changes to how the output is analyzed on both the code cache and object heap tests to parse this new form.

[AlenBadel]

Just to also note, within the object heap Z/OS specifies that pageable 1M large pages should be used. Would it not be more performant to use OMRPORT_VMEM_PAGE_FLAG_PAGEABLE_PREFERABLE instead which ensures that 1M fixed large pages are used if pageable are not configured, rather than downgrading the page size?
@dmitripivkine

https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L145

[dmitripivkine]

https://github.com/eclipse/omr/blob/637f10724dd3de1d4845bd2bc38d9c25456941f3/gc/base/GCExtensionsBase.cpp#L145

In the code you mentioned GC requests 1m pageable as a best choice. Further down it is up to Port Library code to select best choice of pages available. Usually Port Library code assumes that previous element of pages array is a best choice. So for ZOS the array should be [4k non-pageable, 1m non-pageable, 1m pageable, 2g non-pageable]. So previous element should be best choice and it is 1m non-pageable. We should double check that code does this as well as order of the pages is as I mentioned

[gita-omr]

I looked at the design document once again and I have the following comments:

1) I think it still mentions -XX:zOSLargePagesObjectHeap=<pageable|nonpageable>although it was decided to use -XX:zOSLargePagesObjectHeapType=<type>.
2) I5 (mapping of -Xlp) still refers to -XX:+UseNonpageableLargePagesObjectHeap.
3) Are we sure-XX:+/-UsePageableLargePagesObjectHeap could not be used instead?

[AlenBadel]

Thanks for catching that Gita!

Are we sure -XX:+/-UsePageableLargePagesObjectHeap could not be used instead?

We need a tri-state solution. The three cases are as follows:

• (Default) Object heap will attempt to use pageable 1M large pages, but will fall back to 1M fixed, and below if pages are not configured. We tie this behaviour to PREFER_PAGEABLE.
• User may only want Pageable Large Pages.
• User may only want Fixed Large Pages.

Now that I look at it, this raises another minor concern. What if the user specified they wanted pageable large pages, then wants to go back to the default behaviour. This wouldn't be possible without removing the -XX:zOSLargePagesObjectHeapType=pageable argument, and this doesn't fit in the OpenJ9 global option specification. I believe we need add a default scenario so the user is able to overwrite the previous.

I.e -XX:zOSLargePagesObjectHeapType=<pageable|nonpageable|default>

Hence the user in this case could have the following to overwrite the page type:
-XX:zOSLargePagesObjectHeapType=pageable, -XX:zOSLargePagesObjectHeapType=default

[gita-omr]

@pshipton @DanHeidinga @dmitripivkine could you please comment on the proposal from @AlenBadel above?

[DanHeidinga]

This feels like a rat hole. In all years I've worked on the openj9 code base, I don't recall any user issues ever specifying this level of detail about large page settings. We had an agreed on design, let's proceed with it.

If a user shows up with a valid complaint about the situation Alen suggests above, then we'll deal with it at that time. I don't expect it to ever happen though.

[AlenBadel]

@SueChaplain As an FYI, I've finalized the design doc that can be found here: https://github.com/eclipse/openj9/issues/8671#issue-571666508. I will be making doc changes to cover the options we're adding, and modifying the docs of the -Xlp* options that are currently supported. The spreadsheet outlines the changes that are required.