Openj9: openjdk "1.8.0_242": JVMCFRE166 Duplicate entries in LocalVariableTypeTable

[cznewbold]

Java -version output

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
Eclipse OpenJ9 VM (build openj9-0.18.1, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20200122_511 (JIT enabled, AOT enabled)
OpenJ9 - 51a5857d2
OMR - 7a1b0239a
JCL - 8cf8a30581 based on jdk8u242-b08)

Summary of problem

code failing with this error
java.lang.ClassFormatError: JVMCFRE166 Duplicate entries in LocalVariableTypeTable are not allowed; class=com/greenhat/server/container/server/logging/orm/LogEntry, offset=0

attaching the log file

Diagnostic files

If a crash (gpf, assert, abort, etc) or OutOfMemory condition, please provide the diagnostic files produced (javacore, Snap, jitdump, core). The smaller files can be attached to this Issue. The core should be compressed and made available via a file sharing service (Box, Google Drive, etc). If there are privacy concerns please direct email the files to an OpenJ9 committer.

FYI the stderr console will contain messages which describe the location of the diagnostic dump files, similar to the following.

JVMDUMP039I Processing dump event "abort", detail "" at 2019/03/04 01:58:02 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/cmdLineTest_gpTest_0/core.20190304.015802.21708.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/cmdLineTest_gpTest_0/core.20190304.015802.21708.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/cmdLineTest_gpTest_0/javacore.20190304.015802.21708.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/cmdLineTest_gpTest_0/javacore.20190304.015802.21708.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/cmdLineTest_gpTest_0/Snap.20190304.015802.21708.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/cmdLineTest_gpTest_0/Snap.20190304.015802.21708.0003.trc
JVMDUMP007I JVM Requesting JIT dump using '/home/jenkins/cmdLineTest_gpTest_0/jitdump.20190304.015802.21708.0004.dmp'
JVMDUMP010I JIT dump written to /home/jenkins/cmdLineTest_gpTest_0/jitdump.20190304.015802.21708.0004.dmp
JVMDUMP013I Processed dump event "abort", detail "".

OutOfMemoryError: Java Heap Space

For a repeatable OutOfMemory condition showing Java heap space, enable verbose GC collection and provide the log.
--verbose:gc writes the logging to stderr

• -Xverbosegclog writes the logging to a single or multiple files

[DanHeidinga]

@theresa-m Can you take a look at this? I think you had a PR that updated our behaviour recently...

[theresa-m]

I'll take a look.

[theresa-m]

Hi @cznewbold are you able to provide any details for how this error can be reproduced?

I've been going through the Java Virtual Machine Specifications for deviations in OpenJ9's behavior and trying to reproduce a test class locally but I haven't been successful.

One idea to narrow it down would be to isolate each method in LogEntry to figure out which one is having issues with loading while making sure to generate debug info when compiling (javac -g). Each LocalVariableTypeTable is tied to a specific method.

[cznewbold]

hi @theresa-m
the info i got from the developer is that we are using OpenJPA 2.4.2
and our code was working with openjdk 1.8.0_222
but is failing on 1.8.0_242

openjdk is now checking for duplicate entries in LocalVariableTypeTable.
it was not checking for this in the earlier openjdk version

so the question is, can this (probably correct) check be overridden with a switch ?

[pshipton]

Does it also fail with a Hotspot build?
https://adoptopenjdk.net/releases.html

[cznewbold]

i am working on building and testing our product with the hotspot build

what is the difference between the hotspot and regular builds?
is there any way to tell them apart. the java -version / -fullversion does not
mention 'hotspot'

[pshipton]

The Hotspot -version from an AdoptOpenJDK build looks like the following. Although it doesn't mention Hotspot, it is different from the OpenJ9 -version output which does include "OpenJ9". Beyond that you'd have to know where the build comes from and what options were used to create it.

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-202003241709-b06)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b06, mixed mode)

[pshipton]

We're past Milestone 2 for the 0.20.0 release, moving this forward to the next release.

[cznewbold]

finally had time to try this using the hotspot jvm, but alas, our product is using the jre/bin/j9vm files, which don't exist in the hotspot builds

[pshipton]

There is no jre/bin/j9vm directory on Linux, do you mean jre/lib/amd64/j9vm, or did you switch to Windows?

I'm assuming the product is using libjvm.so? You could try copying the jre/bin/server directory in the Hotspot build to j9vm and see if that is enough of a workaround to run the test.

[pshipton]

Is the problem resolved by using the OpenJ9 JVM option -Xshareclasses:none?

[cznewbold]

we were able to try the hotspot jvm on linux to see if the original error
java.lang.ClassFormatError: JVMCFRE166 Duplicate entries in LocalVariableTypeTable are not allowed;
went away.
it appears that the error went away using the hotspot jvm. there were other errors,
which i assume are due to the hotspot jvm being different, but that specific error did not show in the log

i need to get another build in order to try the -Xshareclasses:none option.

[cznewbold]

tried the -Xshareclasses:none option
it also caused the JVMCFRE166 Duplicate entries error to go away

[pshipton]

Thanks. There are a couple other options you can try if possible to narrow it down. They should be tried separately, and without specifying -Xshareclasses:none.
-XX:-ShareAnonymousClasses
-XX:-ShareUnsafeClasses

[pshipton]

btw, the -Xshareclasses:none was new information we didn't know at the time I asked to try with Hotspot.

[cznewbold]

we tried our code with each of the above options, separately.
neither option fixed the error. the duplicate entries error occurs when those are set

we used those options exactly as shown, with the two Xs

thanks

[pshipton]

What is possible to help diagnose this? Can you send something we can use to reproduce the problem? If not, will you provide a core file from the time the exception occurs?
-Xdump:system+java:events=systhrow,filter=java/lang/ClassFormatError

[pshipton]

We discovered more information.

• The problem is not related to the shared classes cache. The reason it occurs when shared classes is enabled is due to the shared cache storing debug information from the class. If the shared cache is enabled but created with -Xnolinenumbers (https://www.eclipse.org/openj9/docs/xlinenumbers/), the problem should not occur because the debug information from the class will not be parsed and verified.
• Likely the problem can be created with Hotspot when debug is enabled (-Xrunjdwp:transport=dt_socket,address=8888,server=y,suspend=n), meaning it's a problem with the class file and not with OpenJ9.

[pshipton]

Updated too soon. The latest information indicates this is a bug in OpenJ9. We'll update when a fix is available.

[pshipton]

@cznewbold a fix is merged, nightly builds tomorrow night at AdoptOpenJDK.net should contain the fix.

[cznewbold]

tried to verify with
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u252-b09.1_openj9-0.20.0
and
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u252-b09.2_openj9-0.20.0
but there are no linux images

[DanHeidinga]

The fix is in the nightly builds which can be found here - https://adoptopenjdk.net/nightly.html?variant=openjdk8&jvmVariant=openj9

[pshipton]

For the record, although they don't contain the fix, the linux builds for the last release are here
https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/tag/jdk8u252-b09_openj9-0.20.0

For a listing of the various builds, which platforms they contain and why, see https://github.com/eclipse/openj9/releases/tag/openj9-0.20.0

[cznewbold]

we ran a test against the nightly builds, and it resolves the problem we were having. we did not have to specify any extra jvm options