Openj9: Create a Large Heap (noncompressedrefs) build

[DanHeidinga]

OpenJ9 supports both compressedrefs and non-compressedrefs packages. Currently, the project only builds and tests the compressedrefs package.

This is a problem for anyone wanting a heap larger than ~57 gig and limits adoption of OpenJ9 for projects that need large heaps.

The long term solution is outlined in https://github.com/eclipse/openj9/issues/643 but we need a more immediate way to meet this need.

Proposal

• Create a "Large Heap" OpenJ9 package for the non-compressedrefs mode to support users that need > ~57 gig heaps.
• Enable building this package on all platforms but make it optional.
• Due to machine limitations, enable this package for compile and test:
  
  
  
  • OpenJ9: pLinux or Windows (depending on machine availability)
  
  
  • Adopt: Enable on xLinux
  
  
• Support this for JDK8 and newer releases (JDK10+)

Next steps:

• [x] Remove any hardcoded references to compressedrefs from OpenJ9 and Extensions makefiles
• [x] Update redirector to generate reasonable messages pointing to the other package when using -Xcompressedrefs/-Xnoncompressedrefs
  
  
  
  • [x] Update build pipelines here to build the non-compressed package #1884
  
  
  • [x] Add it to the OMR promotion criteria #1886
  
  
  • [x] Determine what level of testing here for the non-compressed package
  
  
  • [x] Work with Adopt to get this package built / tested there as well
  
  

Related: https://github.com/eclipse/openj9/issues/479 https://github.com/ibmruntimes/openj9-openjdk-jdk9/issues/20

I'm sure I've missed details here so committers please feel free to edit this to add details I've missed.

[mstoodle]

I hate to be the bearer of hard questions, but do we have any open machines capable of testing a heap that large?

Or do we make an argument that this build just "works that way" and so the actual heap size isn't important for testing? Even if we accept that philosophy, we should still validate in some way that this build can allocate a heap that large. Maybe we just let some job allocate a big heap and then allocate enough big objects to occupy that heap to verify no OOM and just let it thrash on a machine without that much physical memory as a proof of "concept"? Better ideas certainly welcome!

[DanHeidinga]

this build just "works that way" and so the actual heap size isn't important for testing?

@mstoodle The testing should focus on validating the behaviour of the runtime in this mode not on the size of the heap actually allocated. The heap size limitation of compressed is due to the shifting of pointers. No shifting in non-compressedrefs = no limit besides size of pointers / hw capabilities.

We'll have a committer do a one-off test of the build on a machine with sufficient memory to validate the ability to allocate large heaps and then rely on the way the underlying tech works.

[mstoodle]

Classic white box testing reasoning, got it :) .

I am not disagreeing, because the code certainly does work that way (at least at this point in time) so doing much more than what you've, uh, dictated, probably doesn't provide additional testing value from a technical standpoint.

But it still feels a bit weird to rely on one manually instigated test (per platform) as demonstration that this whole other build (per platform) we created specifically for large heaps actually works on large heaps :) .

[DanHeidinga]

Some notes on where to change for noncompressedrefs build: https://github.com/eclipse/openj9/issues/479#issuecomment-346047636

@theresa-m is looking into this.

[theresa-m]

Two PRs are in for the first checkbox: #1773 and https://github.com/ibmruntimes/openj9-openjdk-jdk9/pull/148. The extensions PR is dependent on the OpenJ9 PR. The extensions for Java 8, 10, 11 will also need updating but the changes will be almost exactly the same so waiting for some feedback on 9 first :)

[DanHeidinga]

Reopening as this isn't complete yet - great first steps though!

[DanHeidinga]

@AdamBrousseau Which platforms do we have the most build / test capacity on?

We'd like to enable a non-compressedrefs spec in our builds on one platform. Maybe pLinux to start?

It'll need PR builder syntax tweaks to enable selecting compressed/noncompressed packages

[theresa-m]

Sorry.. thought my rewording the "fixes" reference would prevent the issue from automatically closing! @DanHeidinga The PR merged was actually for the second checkbox. #1773 and https://github.com/ibmruntimes/openj9-openjdk-jdk9/pull/148 are for the first

[AdamBrousseau]

We have 4 of every platform so the only difference will be compile/test times. Valhalla-nestmates is being added on pLinux (to start) so maybe want to start this out on zLinux?

Looking at the nightly build, which runs compile and functional sanity, extended in serial, eyeball average build times:

• AIX
  
  
  
  • 8: 3.5h
  
  
  • 9: 1.5h
  
  
  • 10: 4h
  
  
• zLinux
  
  
  
  • 8: 2.5h
  
  
  • 9: 4h
  
  
  • 10: 4h
  
  
• pLinux
  
  
  
  • 8: 2.5h
  
  
  • 9: 3h
  
  
  • 10: 3h
  
  

For the PR trigger syntax, I'll give it some thought. For nestmates we came up with Jenkins test sanity plinux jdk11-nestmates

[DanHeidinga]

Thanks @theresa-m . I've updated the checkboxes

[DanHeidinga]

Or on our new xLinux boxes :)

[AdamBrousseau]

Yes that is another option. I didn't mention it because we have to data yet on how fast those will be. Same with Windows, although the first few compiles on windows are the fastest of any of our platforms (<20min). Likely something to do with those machines being 8CPU and all the others being 4.

[DanHeidinga]

As the requests for this have all been x86 so far, let's try out the new xLinux boxes for this. It'll also match the package we hope to have Adopt build.

[smlambert]

FYI @llxia @renfeiw - this has implications of work required on test side, given there is special handling of compressed/noncompressedref commandline options.

[llxia]

In test, we check SPEC. If it contains _cmprssptrs, -Xcompressedrefs is used for testing. Otherwise, -Xnocompressedrefs is used.
https://github.com/eclipse/openj9/blob/c4d684500d054d0a5d11e0035cce081ab532e564/test/TestConfig/openj9Settings.mk#L33

As long as we have correct SPEC, then no modification is needed on test side.

[theresa-m]

@DanHeidinga prs for the first checkbox were merged at the beginning of the week for java 8 - 11.

[DanHeidinga]

Update build pipelines here to build the non-compressed package

@AdamBrousseau Can you add nightly builds for the non-compressedrefs package on xLinux and add new regexs to the PR builder for the non-compressed package?

[AdamBrousseau]

builds for the non-compressedrefs package on xLinux

Sure. Going to block this on #1873 as we have little drive space on our xLinux machines

[SueChaplain]

@DanHeidinga Will we be able to claim non-comp refs builds available for 0.9.0, or any progress statement please?

[pshipton]

There is a non-compressed refs build for xLinux. #1917 is blocking turning it on in the OpenJ9 nightlies.

[theresa-m]

I've created an issue for tracking step 6: getting noncompressed packages built & tested at AdoptOpenJDK https://github.com/AdoptOpenJDK/openjdk-build/issues/341

[theresa-m]

I'm writing a blog post to show how to compile non-compressedrefs builds, would anyone object to me adding a note about the configure option into the OpenJ9 website build instructions as well? It seems to me like a useful place to have this information.'

edit: it exists https://github.com/eclipse/openj9-website/pull/93

[SueChaplain]

Please see my comments on the PR. The aim of the build page is to provide the "simplest" set of build steps to make it look easy. So my suggestion would be to add it to the detailed build instructions, with maybe a one liner "If you need a heap size > ....."

[pshipton]

I assume "detailed build instructions" refers to the files in https://github.com/eclipse/openj9/tree/master/buildenv

[DanHeidinga]

XL package is being built on linux x86-64 at Adopt in the nightly builds: https://adoptopenjdk.net/nightly.html?variant=openjdk8-openj9

Marking this as complete