Openjdk-tests: jdk_util fails in TimSortStackSize2 when run with -UseCompressedOops

[adamfarley]

Describe the bug
For some unknown reason, 3 days ago we stopped running jdk_utils with +UseCompressedOops and started running it with -UseCompressedOops.

When we run jdk_util on jdk8 hotspot with -UseCompressedOops, TimSortStackSize2.java fails with this:

[2021-02-24T18:49:13.217Z] java.lang.OutOfMemoryError: Java heap space
[2021-02-24T18:49:13.217Z]  at TimSortStackSize2.createArray(TimSortStackSize2.java:164)
[2021-02-24T18:49:13.218Z]  at TimSortStackSize2.doTest(TimSortStackSize2.java:59)
[2021-02-24T18:49:13.218Z]  at TimSortStackSize2.main(TimSortStackSize2.java:43)
[2021-02-24T18:49:13.218Z]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2021-02-24T18:49:13.218Z]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2021-02-24T18:49:13.218Z]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-02-24T18:49:13.218Z]  at java.lang.reflect.Method.invoke(Method.java:498)
[2021-02-24T18:49:13.218Z]  at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2021-02-24T18:49:13.218Z]  at java.lang.Thread.run(Thread.java:748)

This occurs on most platforms, including:
Mac: https://trss.adoptopenjdk.net/output/test?id=6036cf3a5730424dbc8f9ccf
Windows: https://trss.adoptopenjdk.net/output/test?id=6036b0105730424dbc8f81d8
and xLinux: https://trss.adoptopenjdk.net/output/test?id=6036bc195730424dbc8f8d1e

To Reproduce
https://ci.adoptopenjdk.net/job/Grinder/parambuild/?TARGET=jdk_util&SDK_RESOURCE=upstream&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=0&EXTRA_DOCKER_ARGS=&VENDOR_TEST_DIRS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&PLATFORM=x86-64_linux&KEEP_REPORTDIR=true&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&JDK_VERSION=8&USER_CREDENTIALS_ID=&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptopenjdk%2Fopenjdk-jdk8u&RELEASE_TAG=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk8u%2Fjdk8u-linux-x64-hotspot&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=dev&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&NUM_MACHINES=&OPENJDK_SHA=&BUILD_LIST=openjdk&UPSTREAM_JOB_NUMBER=923&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None

Expected behavior
Unknown. I don't know why we weren't running this test with compressedrefs disabled on Hotspot before, and I don't know what change has resulted in us not running the compressed refs enabled version of the test anymore.

Additional context

The plan:

• [x] Verify whether the test fails on later versions of the jdk.
• [x] Create a table that shows results for different JDK versions, compressedrefs enablement, and platforms.
• [x] See if the test fails if a jdk8 build is used to run a later version of the test. Visa versa.

[llxia]

With testing relying on TKG to detect spec/platform https://github.com/AdoptOpenJDK/TKG/pull/155, TKG sets spec to osx_x86-64 for mac in this case (It was set to osx_x86-64_cmprssptrs before). As a result, a different mode -XX:-UseCompressedOops is used.
TKG behavior is correct. Once we fully switch to mixedref, we are expecting to run both -XX:-UseCompressedOops and -XX:+UseCompressedOops for openjdk tests.

FYI @renfeiw

[adamfarley]

From llxia's comment, I'm reading that it doesn't matter what caused this sudden flip in the modes we're running (at least from the perspective of this issue), as we'll eventually be running with CR turned both off and on.

So, TRSS test results tell us that the issue is not being seen on jdk11 or higher (Example), so I'm comparing the test code to see what's going on.

Edit: The relevant test code difference seems clear. On JDK11, the test code specifies a bigger heap if compressedOops is both non-null and false. Will check to see if a similar change on 8 fixes the issue.

[smlambert]

related: https://github.com/AdoptOpenJDK/openjdk-tests/issues/2138#issuecomment-788784487

[adamfarley]

Re 2138: I'm not sure containers are causing this problem, as I'm able to reproduce the issue locally without using a container. Enabling compressedOops makes the problem go away.

Since jdk11's version of this test (see this bug) sets the Xms to 770m and the Xmx to 1540m in the event of a compressedOops-disabled run, I tried running the test locally with these parameters with compressedOops on and off (on JDK8). Test passed both times. It's not as nuanced as the jdk11 fix, where it only increases the xms/xmx values for compressedOops enabled runs, but it works and it's simple.

Running a multi-platform grinder to verify the fix: https://ci.adoptopenjdk.net/job/Grinder/7408/console

If this works, I recommend excluding the test for now and contributing the fix upstream.

Shelley, Lan, this will mean that this unit test will always run with Xms 770 and Xmx 1540 (which is what jdk11+ seems to use for CR-disabled anyway). Any concerns?

Edit: Every platform tested in that grinder passed the test with the xms and xms settings hard-coded to the compressed-refs-specific values used in jdk11.

Platforms were x86-64_linux, ppc64_aix, s390x_linux, aarch64_linux, ppc64le_linux, x86-64_mac, x86-64_windows, and x86-32_windows.

[andrew-m-leonard]

@adamfarley i've noticed this issue as well, I think we need to backport https://bugs.openjdk.java.net/browse/JDK-8199265 to jdk-8. Do you want me to do that?

[adamfarley]

@andrew-m-leonard - I saw that bug, but it depends on a bunch of other code changes too in order to get that particular code change working. It's simpler to just increase the heap size manually on the test. I'm trying to run a grinder where that increased heap size is set in the playlist, but the grinder seems to be having difficulties downloading builds.

https://ci.adoptopenjdk.net/job/Grinder/7487/console

[andrew-m-leonard]

@adamfarley sounds good to me

[smlambert]

@sophia-guo - see links to @adamfarley 's Grinder jobs... wondering if recent PR https://github.com/AdoptOpenJDK/openjdk-tests/pull/2320 is causing the issue (Grinders are a jdk8 build, and look to try and download testimage, despite --test_images_required false)

08:32:02  + ./openjdk-tests/get.sh -s /home/jenkins/workspace/Grinder -t /home/jenkins/workspace/Grinder/openjdk-tests -p x86-64_linux -r nightly -j 8 -i hotspot --openj9_repo https://github.com/eclipse/openj9.git --openj9_branch master --tkg_repo https://github.com/AdoptOpenJDK/TKG.git --tkg_branch master --test_images_required false --debug_images_required false
08:32:02  TESTDIR: /home/jenkins/workspace/Grinder/openjdk-tests
08:32:02  get jdk binary...
08:32:02  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks  https://api.adoptopenjdk.net/v3/binary/latest/8/ea/linux/x64/jdk/hotspot/normal/adoptopenjdk
08:32:06  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks  https://api.adoptopenjdk.net/v3/binary/latest/8/ea/linux/x64/testimage/hotspot/normal/adoptopenjdk
08:32:06  Uncompressing file: adoptopenjdk ...
08:32:06  
08:32:06  gzip: adoptopenjdk: not in gzip format
08:32:06  tar: This does not look like a tar archive
08:32:06  tar: Exiting with failure status due to previous errors

[adamfarley]

Running from upstream: https://ci.adoptopenjdk.net/job/Grinder/7493/

I noticed that modifying the jdk_utils entry in the playlist means that any attempt to run the individual test case in a grinder puts it inside the jdk_custom target, and results in the same oom we're trying to prevent via the modification of jdk_utils.

So I'm going to propose modifying jdk_utils for now so we can run the test, and I'll propose the inside-testcase fix to the jdk8 update folks.

[adamfarley]

Short term fix PR created: https://github.com/AdoptOpenJDK/openjdk-tests/pull/2339

Contacting upstream now for the long term fix: https://bugs.openjdk.java.net/browse/JDK-8263099

[sophia-guo]

Re: https://github.com/AdoptOpenJDK/openjdk-tests/issues/2310#issuecomment-791441021. Look like no testimage created for jdk8 hotspot. https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-x64-hotspot/, which makes sense as jdk8 source and test are under same directory. (openj9 test image is different).

@andrew-m-leonard could you confirm that?

[sophia-guo]

Re: https://github.com/AdoptOpenJDK/openjdk-tests/issues/2310#issuecomment-791441021 test_images_required is used to auto download testimage jar if CUSTOMIZED_SDK_URL is used and specifically for OpenJ9-JDK.

https://github.com/AdoptOpenJDK/openjdk-tests/pull/1692

[adamfarley]

For TimSortStackSize2's long-term upstream fix, Paul Hohensee recommends merging four relevant change sets from after 8u, rather than diverging from the path taken to resolve this in later releases.

Relevant changesets (which, as advised, should be applied in order):
http://hg.openjdk.java.net/jdk-updates/jdk9u/jdk/rev/cce93a64b059
http://hg.openjdk.java.net/jdk/hs/rev/1836bf0c820a
http://hg.openjdk.java.net/jdk/jdk/rev/9b0e2937fac5
http://hg.openjdk.java.net/jdk/jdk/rev/dfde3bb48c03

Giving this a try.

Edit: 3 of those 4 were not clean backports, but I was able to backport enough of them to make the fix work.

https://github.com/AdoptOpenJDK/openjdk-jdk8u/compare/master...adamfarley:oom_upstream_merge_fix

Have updated the community, and will wait on their reply.

Update: The community prefers individual changesets for backporting, to preserve history. So three of those changesets were identified as "key", and divided between the original issues. We now have the reviews we need, and await approval from the jdk8u backport review group.

Follow this issue for further updates: https://bugs.openjdk.java.net/browse/JDK-8263099