Openj9: JVM Crash on jdk15-mac-x64-openj9

[M-Davies]

Java -version output

THIS IS NOT THE JAVA VERSION OF THE FAILURE BELOW. The failure occurred mid build so there's no java version to display. The closest I can get is a build from 30-Jul-2020 so it should be fairly similar:

22:23:55  openjdk version "15" 2020-09-15
22:23:55  OpenJDK Runtime Environment AdoptOpenJDK (build 15+33-202007302055)
22:23:55  Eclipse OpenJ9 VM AdoptOpenJDK (build master-500b10edb, JRE 15 Mac OS X amd64-64-Bit Compressed References 20200730_63 (JIT enabled, AOT enabled)
22:23:55  OpenJ9   - 500b10edb
22:23:55  OMR      - 7b76a160b
22:23:55  JCL      - 88d915c853 based on jdk-15+33)

Summary of problem

JVM crash mid build at Adopt on a jdk15-mac-x64-openj9 nightly build (machine build-macstadium-macos1014-x64-2)

Diagnostic files

Core dump files should be still available on the machine. Open an issue at https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/new to request access to the machine to retrieve the dump files.

22:10:50  Creating java.desktop.jmod
22:11:09  Assertion failed at /Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/build/macosx-x86_64-server-release/vm/compiler/../compiler/compile/J9SymbolReferenceTable.cpp:823: containingClass != NULL
22:11:09  VMState: 0x00050080
22:11:09    failed to get defining class of field ref cpIndex=23 in owning method J9Method=0x15672548
22:11:09  compiling jdk/internal/org/objectweb/asm/MethodWriter.<init>(Ljdk/internal/org/objectweb/asm/SymbolTable;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V at level: warm
22:11:09  
22:11:09  JVMDUMP039I Processing dump event "abort", detail "" at 2020/08/02 22:11:09 - please wait.
22:11:09  JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/core.20200802.221109.68820.0001.dmp' in response to an event
22:11:16  JVMDUMP012E Error in System dump: The core file created by child process with pid = 68886 was not found. Expected to find core file with name "/cores/core.68886"
22:11:16  JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/javacore.20200802.221109.68820.0002.txt' in response to an event
22:11:16  JVMDUMP010I Java dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/javacore.20200802.221109.68820.0002.txt
22:11:16  JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/Snap.20200802.221109.68820.0003.trc' in response to an event
22:11:16  JVMDUMP010I Snap dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/Snap.20200802.221109.68820.0003.trc
22:11:16  JVMDUMP032I JVM requested JIT dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/jitdump.20200802.221109.68820.0004.dmp' in response to an event
22:11:17  Creating java.prefs.jmod
22:11:17  Creating java.net.http.jmod
22:11:17  Creating java.rmi.jmod
22:11:18  Creating java.scripting.jmod
22:11:18  JVMDUMP010I JIT dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/make/jitdump.20200802.221109.68820.0004.dmp
22:11:18  JVMDUMP013I Processed dump event "abort", detail "".
22:11:19  make[3]: *** [/Users/jenkins/workspace/build-scripts/jobs/jdk15/jdk15-mac-x64-openj9/workspace/build/src/build/macosx-x86_64-server-release/support/images/jmods/_create_java.logging.jmod_exec.marker] Error 1
22:11:19  make[2]: *** [java.logging-jmod] Error 2
22:11:19  make[2]: *** Waiting for unfinished jobs....

OutOfMemoryError: Java Heap Space

For a repeatable OutOfMemory condition showing Java heap space, enable verbose GC collection and provide the log.
--verbose:gc writes the logging to stderr

• -Xverbosegclog writes the logging to a single or multiple files

[dmitripivkine]

@andrewcraik FYI: looks like jit compilation problem based on vmState 0x00050080

[andrewcraik]

@rpshukla could you have a look - FYI @liqunl

[liqunl]

This is the same issue @Leonardo2718 looked at.

[Leonardo2718]

Yes, this looks like the same issue as https://github.com/eclipse/openj9/issues/9416 (which is good because I haven't been able to reproduce that one at all).

[Leonardo2718]

I've opened, https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/1490. If I can get the build and core files quickly, hopefully, I can make more progress than I've been able to so far.

[austin0]

Have just seen this again on build-macstadium-macos1014-x64-1 (https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk15u/job/jdk15u-mac-x64-openj9-macosXL/18/console) @Leonardo2718 .

21:55:25  Assertion failed at /Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/build/macosx-x86_64-server-release/vm/compiler/../compiler/compile/J9SymbolReferenceTable.cpp:823: containingClass != NULL
21:55:25  VMState: 0x00050080
21:55:25    failed to get defining class of field ref cpIndex=12 in owning method J9Method=0x4004c060
21:55:25  compiling jdk/internal/org/objectweb/asm/MethodWriter.<init>(Ljdk/internal/org/objectweb/asm/SymbolTable;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V at level: warm
21:55:25  
21:55:25  JVMDUMP039I Processing dump event "abort", detail "" at 2020/08/26 21:54:55 - please wait.
21:55:25  JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/core.20200826.215455.3561.0001.dmp' in response to an event
21:55:25  JVMDUMP012E Error in System dump: The core file created by child process with pid = 3585 was not found. Expected to find core file with name "/cores/core.3585"
21:55:25  JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/javacore.20200826.215455.3561.0002.txt' in response to an event
21:55:25  JVMDUMP010I Java dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/javacore.20200826.215455.3561.0002.txt
21:55:25  JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/Snap.20200826.215455.3561.0003.trc' in response to an event
21:55:25  JVMDUMP010I Snap dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/Snap.20200826.215455.3561.0003.trc
21:55:25  JVMDUMP032I JVM requested JIT dump using '/Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/jitdump.20200826.215455.3561.0004.dmp' in response to an event
21:55:25  JVMDUMP010I JIT dump written to /Users/jenkins/workspace/build-scripts/jobs/jdk15u/jdk15u-mac-x64-openj9-macosXL/workspace/build/src/make/jitdump.20200826.215455.3561.0004.dmp
21:55:25  JVMDUMP013I Processed dump event "abort", detail "".

[Leonardo2718]

Unfortunately, it appears that core files can't be produced on MacOS (see https://github.com/AdoptOpenJDK/openjdk-build/issues/1778). Without a core file, I don't really have a way of investigating this issue.

[adamfarley]

Is this issue related to #9291? They look similar.

Also, this issue is still happening. Link below.

https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-mac-x64-openj9-macosXL/682/console

[Leonardo2718]

I've already looked into #9291 but that issue has never been reproduced AFAIK and the core file I got didn't contain any useful information. I haven't been able to reproduce the failure myself so far either, so I don't have anything to look at.

There really is nothing I can do until I get a core file.

[pshipton]

Closing as a dup of #9416.

There is another occurrence https://ci.eclipse.org/openj9/job/Build_JDK15_x86-64_mac_Nightly/71/, and a core file in https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Build_JDK15_x86-64_mac_Nightly/71/Build_JDK15_x86-64_mac_Nightly-71-20201006-235858-diagnostics.tar.gz

reported in https://github.com/eclipse/openj9/issues/9416#issuecomment-704945035