graalvm 20.0.0 crash on jdk 11

[pmlopes]

Describe GraalVM and your environment :

• GraalVM version or commit id if built from source: 20.0.0
• CE or EE: CE
• JDK version: JDK11
• OS and OS Version: linux
• Architecture: amd64 AND aarch64
• The output of java -Xinternalversion:

OpenJDK 64-Bit Server VM (11.0.6+9-jvmci-20.0-b02) for linux-amd64 JRE (11.0.6+9-jvmci-20.0-b02), built on Jan 20 2020 20:06:41 by "buildslave" with gcc 7.3.0

Have you verified this issue still happens when using the latest snapshot?
Nope. This is the latest stable release.

Describe the issue
The VM crashes while running es4x unit tests.

Code snippet or code repository that reproduces the issue

git clone https://github.com/reactiverse/es4x.git
cd es4x
mvn test

Steps to reproduce the issue

git clone https://github.com/reactiverse/es4x.git
cd es4x
mvn test

Expected behavior
Test run. They should pass as they pass on graal jdk8 or graal 19.3.1

See travis logs here:

https://travis-ci.com/reactiverse/es4x/jobs/288871429
https://travis-ci.com/reactiverse/es4x/jobs/288871430

Additional context

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000ffff8bee23e0, pid=2797, tid=2817
#
# JRE version: OpenJDK Runtime Environment GraalVM CE 20.0.0 (11.0.6+9) (build 11.0.6+9-jvmci-20.0-b02)
# Java VM: OpenJDK 64-Bit Server VM GraalVM CE 20.0.0 (11.0.6+9-jvmci-20.0-b02, mixed mode, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x6f13e0]  JfrEventClassTransformer::on_klass_creation(InstanceKlass*&, ClassFileParser&, Thread*)+0x6f8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/travis/build/reactiverse/es4x/es4x/hs_err_pid2797.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

[chumer]

Could you attache the /home/travis/build/reactiverse/es4x/es4x/hs_err_pid2797.log please?

[pmlopes]

Well, I didn't save that one, but i can reproduce it 100%, so this is a local run on my laptop.

hs_err_pid29078.log

[pmlopes]

Small update! It's not related to es4x! :tada: It doesn't crash on my laptop if I disable jacoco.

<!--      <plugin>-->
<!--        <groupId>org.jacoco</groupId>-->
<!--        <artifactId>jacoco-maven-plugin</artifactId>-->
<!--        <version>0.8.5</version>-->
<!--        <executions>-->
<!--          <execution>-->
<!--            <goals>-->
<!--              <goal>prepare-agent</goal>-->
<!--            </goals>-->
<!--          </execution>-->
<!--          <execution>-->
<!--            <id>report</id>-->
<!--            <phase>test</phase>-->
<!--            <goals>-->
<!--              <goal>report</goal>-->
<!--            </goals>-->
<!--          </execution>-->
<!--        </executions>-->
<!--      </plugin>-->

[pmlopes]

So this tells us that it's not a polyglot issue, i guess but a vm agent issue...

[chumer]

At first sight. I think this might happen in general with any agent configured.

[chumer]

It is related to the JFR support we added in 20.0.

[gilles-duboscq]

This looks like JDK-8232997

[gilles-duboscq]

This is fixed upstream and will land in JDK 11.0.7 and GraalVM 20.0.1 for the April CPU.
/cc @dougxc let's remember to test this with a 11.0.7-based build.

[tzezula]

You can try to disable JFR by setting the jfr.unsupported.vm system property to true.
https://hg.openjdk.java.net/jdk/jdk11/file/1ddf9a99e4ad/src/jdk.jfr/share/classes/jdk/jfr/internal/JVMSupport.java#l46

[gilles-duboscq]

Thanks @tzezula, in the case an agent is really important it seems setting -Djfr.unsupported.vm=true is a good workaround.

[albogdano]

Probably related to this issue (same version GraalVM 20.0.0/jdk11):
The first time I ran a large batch import job this happened:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fbf0bc91f8c, pid=12885, tid=13249
#
# JRE version: OpenJDK Runtime Environment GraalVM CE 20.0.0 (11.0.6+9) (build 11.0.6+9-jvmci-20.0-b02)
# Java VM: OpenJDK 64-Bit Server VM GraalVM CE 20.0.0 (11.0.6+9-jvmci-20.0-b02, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x18ef8c]
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/alexb/Desktop/paraio/hs_err_pid12885.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Then I ran it again and I got this:

JDWP exit error AGENT_ERROR_NULL_POINTER(201): thread list corrupted [:2148]
FATAL ERROR in native method: JDWP thread list corrupted, jvmtiError=AGENT_ERROR_NULL_POINTER(201)

hs_err_pid12885.log

[gilles-duboscq]

@albogdano i'm not sure that's related.
In your hs_err it looks like there is a problem with the JDWP agent when starting a JVMCI compiler thread.
Some possible issues:

• it's running in libgraal mode so that JavaThread might look a little unusual?
• there's been a GC between the creation of the thread and the time the thread creation was reported to the agent, maybe there's a naked oop somewhere?

/cc @tkrodriguez, @dougxc

[axel22]

@albogdano Could you please post the exact steps that triggered that error? Did you also run mvn test on the es4x project? Thank you.

[albogdano]

@axel22 Actually my crash is not related to the es4x project at all but unfortunately I cannot reproduce it now. Apparently it only happened once.
I'm running the latest GraalVM CE 20.0.0 JDK 11. Steps are:

1. I run a Spring Boot application I do a full backup of the database (here's the code executed)
2. The JVM crashed when I wanted to restore the backup by uploading the zip file to the application where it gets unzipped and loaded on the fly using streams (restore code here)
   Now I execute the same steps but there's no crash.

[axel22]

@albogdano Ok, thank you for detailing the steps. It seems that the error is non-deterministic, and depends on the precise timing of the lifecycles of the JVMCI compiler threads in the JVM.
I will keep trying to reproduce it.

[gilles-duboscq]

The JDWP crash looks like #2320

[mavocitrix]

I just encountered the same/similar error on OpenJDK 64-Bit Server VM GraalVM CE 20.2.0 (11.0.8+10-jvmci-20.2-b03. In this case it was also caused by jacoco plugin. When I removed it, build worked fine.

The proposed workaround with -Djfr.unsupported.vm=true worked fine with jacoco enabled.

Attaching thread dump:
hs_err_pid30776.log

[dougxc]

Since OpenJDK 64-Bit Server VM GraalVM CE 20.2.0 (11.0.8+10-jvmci-20.2-b03) includes JDK-8232997, there must be some other JFR problem here.
Hopefully we can get some non-Windows hs-err logs attached to this issue to better understand where the problem may be. The Windows hs-err log is not very helpful when it comes to the native frames:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x374d78]
V  [jvm.dll+0x3744d9]
V  [jvm.dll+0x37488c]
V  [jvm.dll+0x4e8a3a]
V  [jvm.dll+0x6d4389]
V  [jvm.dll+0x3ce468]
V  [jvm.dll+0x3d3b68]
C  [java.dll+0x1b41]

[tkrodriguez]

Decoding that stack using the jvm.map file gives this which I think matches well with the existing crash. The output converted hex to decimal and the last two number of the ranges of the symbol itself.

3620216 ?on_klass_creation@JfrEventClassTransformer@@SAXAEAPEAVInstanceKlass@@AEAVClassFileParser@@PEAVThread@@@Z 3618368 3621168
3618009 ?get_u2@StackMapStream@@QEAAGPEAVThread@@@Z 3615248 3618368
3618956 ?on_klass_creation@JfrEventClassTransformer@@SAXAEAPEAVInstanceKlass@@AEAVClassFileParser@@PEAVThread@@@Z 3618368 3621168
5143098 ?create_from_stream@KlassFactory@@CAPEAVInstanceKlass@@PEAVClassFileStream@@PEAVSymbol@@PEAVClassLoaderData@@VHandle@@PEBV2@PEAV?$GrowableArray@VHandle@@@@PEAVThread@@@Z 5142416 5143344
7156617 ?resolve_from_stream@SystemDictionary@@SAPEAVInstanceKlass@@PEAVSymbol@@VHandle@@1PEAVClassFileStream@@PEAVThread@@@Z 7156096 7156800
3986536 ?find_class_from_class_loader@@YAPEAV_jclass@@PEAUJNIEnv_@@PEAVSymbol@@EVHandle@@2EPEAVThread@@@Z 3983392 3987328
4008808 JVM_DefineClassWithSource 4008528 4008928