Temurin-build: problem jdk11.0.2+9 G1BarrierSetC2::eliminate_gc_barrier JVM crash

[leeharland]

**Platform:**
Host: MacPro6,1 x86_64 3500 MHz, 12 cores, 32G, Darwin 18.0.0

**Architecture:**
JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# V  [libjvm.dylib+0x306154]  G1BarrierSetC2::eliminate_gc_barrier(PhaseMacroExpand*, Node*) const+0x1c4
#

Hi, this is the first time i've filed a JDK bug so apologies if this isnt correct. We have a large server-application where we're seeing this issue, while i dont think I can outline steps to recreate it externally, i am happy to help in whatever way i can. The crash is absolutely reproducible on our side which i hope will help with any proposed solution testing.

There is consistency in the CompileTask line:
com.scibite.termitej.document.parser.ndp.ChunkedDocumentCollection:: (104 bytes)

This is a piece of code that caches a (File) InputStream into a byte []. There is plenty of RAM available and the files are not huge. If i comment out the init method of this class (meaning the input stream is ignored) then the JVM crash is not observed. I am currently looking at whether i can isolate this to more specific lines of code and provide them to you, but the class is nothing fancy, its not doing anything abnormal to my knowledge.

Happy to help more apologies this isnt a simple test I will keep trying to identify the specific line(s) on my side and provide them if possible;

More data ( I can supply full crash files if required)

-------------- S U M M A R Y ------------

Command Line: -Xmx20g -Xmx8g -javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=53499:/Applications/IntelliJ IDEA CE.app/Contents/bin -Dfile.encoding=UTF-8 unittest.support.TermiteTestServer

Host: MacPro6,1 x86_64 3500 MHz, 12 cores, 32G, Darwin 18.0.0
Time: Sat Mar 16 16:41:42 2019 GMT elapsed time: 144 seconds (0d 0h 2m 24s)

--------------- T H R E A D ---------------

Current thread (0x00007ff874983000): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=15619, stack(0x000070000588f000,0x000070000598f000)]

Current CompileTask:
C2: 144429 10479 4 com.scibite.termitej.document.parser.ndp.ChunkedDocumentCollection:: (104 bytes)

Stack: [0x000070000588f000,0x000070000598f000], sp=0x000070000598a7b0, free space=1005k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.dylib+0x306154] G1BarrierSetC2::eliminate_gc_barrier(PhaseMacroExpand, Node) const+0x1c4
V [libjvm.dylib+0x53fb0b] PhaseMacroExpand::process_users_of_allocation(CallNode)+0x197
V [libjvm.dylib+0x5402f4] PhaseMacroExpand::eliminate_allocate_node(AllocateNode)+0x1c8
V [libjvm.dylib+0x54551e] PhaseMacroExpand::eliminate_macro_nodes()+0x13c
V [libjvm.dylib+0x231b59] Compile::Optimize()+0x6b3
V [libjvm.dylib+0x2307cf] Compile::Compile(ciEnv, C2Compiler, ciMethod, int, bool, bool, bool, DirectiveSet)+0x8c1
V [libjvm.dylib+0x23296d] Compile::Compile(ciEnv, C2Compiler, ciMethod, int, bool, bool, bool, DirectiveSet)+0x31
V [libjvm.dylib+0x1b12b3] C2Compiler::compile_method(ciEnv, ciMethod, int, DirectiveSet)+0x107
V [libjvm.dylib+0x23d560] CompileBroker::invoke_compiler_on_method(CompileTask)+0x5e0
V [libjvm.dylib+0x23ce04] CompileBroker::compiler_thread_loop()+0x1c8
V [libjvm.dylib+0x71d0fa] JavaThread::thread_main_inner()+0x7e
V [libjvm.dylib+0x60de63] thread_native_entry(Thread*)+0x12b
C [libsystem_pthread.dylib+0x333d] _pthread_body+0x7e
C [libsystem_pthread.dylib+0x62a7] _pthread_start+0x46
C [libsystem_pthread.dylib+0x2425] thread_start+0xd

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000010

[karianna]

Hi @leeharland - Is this an AdoptOpenjDK binary that you are using? Can you also check if this is an issue with an alternative 11.0.2+9 binary from another provider? I'm trying to ascertain if this needs to be reported upstream :-).

[karianna]

Are you able to share the source code of that file? It might also be helpful to run that class file through the JitWatch tool (hosted here on the AdoptOpenJDK GitHub repo) and send in the byte code and machine code for that class/method.

[leeharland]

Thanks yep will try to look into all of that and let you know. Away for a week but will get back to you asap

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Martijn Verburg [e-mail]
Sent: Sunday, March 17, 2019 10:59 am
To: AdoptOpenJDK/openjdk-build
Cc: Lee Harland; Mention
Subject: Re: [AdoptOpenJDK/openjdk-build] problem jdk11.0.2+9 G1BarrierSetC2::eliminate_gc_barrier JVM crash (#967)

Are you able to share the source code of that file? It might also be helpful to run that class file through the JitWatch tool (hosted here on the AdoptOpenJDK GitHub repo). and send in the bye code and machine code for that class / method.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/AdoptOpenJDK/openjdk-build/issues/967#issuecomment-473654776, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABPaDTldqBqJkCBohHezo2EsL9kZvf1Wks5vXiAagaJpZM4b36r4.

[leeharland]

Apologies for the very long delay in responding, i was trying to find the time to do some proper investigation.

The first thing was i tried a bunch of different binaries as suggested. All of these failed .

Adopt: openjdk version "11.0.1" 2018-10-16/OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.1+13) /OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.1+13, mixed mode)
   #  SIGSEGV (0xb) at pc=0x0000000104b0f374, pid=4884, tid=17155
   #
   # JRE version: OpenJDK Runtime Environment (11.0.1+13) (build 11.0.1+13)
   # Java VM: OpenJDK 64-Bit Server VM (11.0.1+13, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
   # Problematic frame:
   # V  [libjvm.dylib+0x30f374]  _ZNK14G1BarrierSetC220eliminate_gc_barrierEP16PhaseMacroExpandP4Node+0x1c4

Oracle: java version "12.0.2" 2019-07-16 / Java(TM) SE Runtime Environment (build 12.0.2+10) /Java HotSpot(TM) 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

   #  SIGSEGV (0xb) at pc=0x00000001038e49d4, pid=5363, tid=22019
   # JRE version: Java(TM) SE Runtime Environment (12.0.2+10) (build 12.0.2+10)
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (12.0.2+10, mixed mode, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
   # Problematic frame:
   # V  [libjvm.dylib+0x2e49d4]  G1BarrierSetC2::eliminate_gc_barrier(PhaseMacroExpand*, Node*) const+0x1c4


Amazon: openjdk version "11.0.4" 2019-07-16 LTS / OpenJDK Runtime Environment Corretto-11.0.4.11.1 (build 11.0.4+11-LTS) /OpenJDK 64-Bit Server VM Corretto-11.0.4.11.1 (build 11.0.4+11-LTS, mixed mode)
   #  SIGSEGV (0xb) at pc=0x000000010b6f72f7, pid=5462, tid=14595
   # JRE version: OpenJDK Runtime Environment (11.0.4+11) (build 11.0.4+11-LTS)
   # Java VM: OpenJDK 64-Bit Server VM (11.0.4+11-LTS, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
   # Problematic frame:
   # V  [libjvm.dylib+0x2f72f7]  _ZNK14G1BarrierSetC220eliminate_gc_barrierEP16PhaseMacroExpandP4Node+0x1c1

Adopt: openjdk version "12.0.2" 2019-07-16 / OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.2+10) /OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.2+10, mixed mode, sharing)
   # JRE version: OpenJDK Runtime Environment (12.0.2+10) (build 12.0.2+10)
   # Java VM: OpenJDK 64-Bit Server VM (12.0.2+10, mixed mode, sharing, tiered, compressed oops, g1 gc, bsd-amd64)
   # Problematic frame:
   # V  [libjvm.dylib+0x2e6d05]  _ZNK14G1BarrierSetC220eliminate_gc_barrierEP16PhaseMacroExpandP4Node+0x1bb

However, the open J9 version worked absolutely fine, no problems at all

Adopt: openjdk version "12.0.2" 2019-07-16 / OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.2+10) /Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.15.1, JRE 12 Mac OS X amd64-64-Bit Compressed References 20190719_142 (JIT enabled, AOT enabled) /OpenJ9   - 0f66c6431 /OMR      - ec782f26 / JCL      - 06c2cc3322 based on jdk-12.0.2+10)

  ==> Worked No crash

Originally i thought it was consistent in the class, but its not, on performing many more iterations it was clear its in different places. However, one thing i did see was our "createTemporaryFile" method coming up a few times (and this is literally new File(java.io.tmpdir,xxx) and thats it). And in other classes such as the original one above, its known to use temp files so may be a red herring or may be something of significance. The reason i say this is that we see this in a client/server application - this is an extreme stress-test unit test where we throw huge volume of data in multiple parallel threads at the server. The server will be creating quite a lot of temporary files (deliberately, pushing it up to 30k+ files to ensure the server is still able to cope) here which is something our other unit tests are not testing, and they don't exhibit this crash.

I hope this is in some way helpful.

[leeharland]

Further to this - i did some more digging and i think i've come up with a reproducible test case. It turns out it was not the temporary files but the logging. We use tinylog in some of the modules and it was here we see the crash. I'd be interested if the attached test breaks on your machine. I have tested this on multiple Mac's and see consistent crashes under the JVM11's outlined above.

Please ignore how totally insane this class is, its not a real piece of our code but just a bizarre combination of things that i ended up with after trying to invent a somewhat similar situation.. even the tiniest change seems to 'fix' the problem and the code runs fine, but this specific combination of silly code seems to break it. Im not sure its the files themselves that have any relevance, when i make just plain objects i get the same crash ... just not as consistently.

Ultimately, it may be that tinylog or this toy code is doing something stupid, but i guess the point is that this causes no crash on any java8, or the openj9 but does in the others.

JVMCrashOnJDK11.java.txt

[karianna]

OK, so I think this is an upstream issue that should get reported at OpenJDK itself. Apologies for bouncing you around! Can you post your findings to the gc-dev mailing list: http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-dev

Thanks!

[leeharland]

np thanks have reported as suggested

[jerboaa]

This is being tracked as https://bugs.openjdk.java.net/browse/JDK-8229016 now, after @leeharland reported it upstream. Thanks!