Graal: "CTW" crashes on AArch64.

Created on 1 Nov 2019  路  9Comments  路  Source: oracle/graal

Running "ctw" task on AArch64 crashes. Here is part of the error log:

12:28:32 CompileTheWorld : iteration 0
12:28:33 #
12:28:33 # A fatal error has been detected by the Java Runtime Environment:
12:28:33 #
12:28:33 #  SIGILL (0x4) at pc=0x0000000800001ac8, pid=11787, tid=11788
12:28:33 #
12:28:33 # JRE version: OpenJDK Runtime Environment (11.0.4+11) (build 11.0.4+11-post- 
Ubuntu-1ubuntu218.04.3)
12:28:33 # Java VM: OpenJDK 64-Bit Server VM (11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed 
mode, sharing, tiered, jvmci, compressed oops, g1 gc, linux-aarch64)
12:28:33 # Problematic frame:
12:28:33 # C  0x0000000800001ac8
12:28:33 #
12:28:33 # Core dump will be written. Default location: Core dumps may be processed with 
"/usr/share/apport/apport %p %s %c %d %P" (or dumping to /home/ent-user/ci- scripts/core.11787)
12:28:33 #
12:28:33 # An error report file with more information is saved as:
12:28:33 # /home/ent-user/ci-scripts/hs_err_pid11787.log
12:28:33 Compiled method (c1)    1208 1206       3       java.util.Arrays::sort (47 bytes)
12:28:33  total in heap  [0x0000ffff790bf190,0x0000ffff790bf908] = 1912
12:28:33  relocation     [0x0000ffff790bf308,0x0000ffff790bf398] = 144
12:28:33  main code      [0x0000ffff790bf3c0,0x0000ffff790bf640] = 640
12:28:33  stub code      [0x0000ffff790bf640,0x0000ffff790bf790] = 336
12:28:33  metadata       [0x0000ffff790bf790,0x0000ffff790bf7d0] = 64
12:28:33  scopes data    [0x0000ffff790bf7d0,0x0000ffff790bf848] = 120
12:28:33  scopes pcs     [0x0000ffff790bf848,0x0000ffff790bf8e8] = 160
12:28:33  dependencies   [0x0000ffff790bf8e8,0x0000ffff790bf8f0] = 8
12:28:33  nul chk table  [0x0000ffff790bf8f0,0x0000ffff790bf908] = 24
12:28:33 #
12:28:33 # If you would like to submit a bug report, please visit:
12:28:33 #   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
12:28:33 #

And the command:

mx --primary-suite-path .../graal/compiler gate -o -t 'CTW:hosted'
bug compiler
Source
picture XiaohongGong

All 9 comments

The error happens with a high probability.

XiaohongGong picture XiaohongGong on 1 Nov 2019

I tried to reproduce your issue. I used the official docker image from ubuntu arm64v8/ubuntu with graal revision 54dcaeba236382fc1fcf96a90a1202fba8020576 and ran CTW for 20 times without any error. For completeness you may provide more detailed hs_err_* file..

root@675443339a1e:~/graal/compiler# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
root@675443339a1e:~/graal/compiler# java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode)
root@675443339a1e:~/graal/compiler# git log -1
commit 54dcaeba236382fc1fcf96a90a1202fba8020576 (HEAD -> master, origin/master, origin/HEAD)
Merge: aee967f1d90 546c77f9d48
Author: Vojin Jovanovic <[email protected]>
Date:   Tue Nov 5 12:13:27 2019 +0000

    [GR-19385] Generate complete stack trace string.

    PullRequest: graal/4818

root@675443339a1e:~/graal/compiler# echo $(seq 20) | xargs -P 4 -n 1 bash -c "mx  gate -o -t 'CTW:hosted'"
gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: Gategate: 05 Nov 2019 15:42:44(+00:00) BEGIN: Gate

gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: CTW:hostedgate: 05 Nov 2019 15:42:44(+00:00) BEGIN: CTW:hosted

gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: Gate
gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: Gate
gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: CTW:hosted
gate: 05 Nov 2019 15:42:44(+00:00) BEGIN: CTW:hosted
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
CompileTheWorld : iteration 0
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
CompileTheWorld : iteration 0
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
CompileTheWorld : iteration 0
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
CompileTheWorld : iteration 0
Monitor starting
Unsupported architecture aarch64
Unsupported architecture aarch64
Unsupported Operating System linux
CompileTheWorld : Waiting for 201631 compiles, just completed 0 compiles
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
CompileTheWorld : Waiting for 189657 compiles, just completed 11974 compiles
Monitor starting
Unsupported architecture aarch64
Unsupported architecture aarch64
Unsupported Operating System linux
CompileTheWorld : Waiting for 201631 compiles, just completed 0 compiles
CompileTheWorld : Waiting for 166197 compiles, just completed 23460 compiles
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
CompileTheWorld : Waiting for 196523 compiles, just completed 5108 compiles
CompileTheWorld : Waiting for 147730 compiles, just completed 18467 compiles
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
.....
--o-o-o-o-o-o-o-o-o-o-o-o-o-o-
CompileTheWorld : Waiting for 7867 compiles, just completed 17125 compiles
CompileTheWorld : Done (28662 classes, 201631 methods, 257892882786 ms elapsed, 7363977 ms compile time, 142312528056 bytes of memory used)
gate: 05 Nov 2019 16:09:46(+04:20) END:   CTW:hosted [0:04:20.405428] [disk (free/total): 18.9GB/25.0GB]
WARNING: Removing scaladacapo:actors from benchmarks because corba has been removed since JDK11 (http://openjdk.java.net/jeps/320)
gate: 05 Nov 2019 16:09:46(+04:20) END:   Gate [0:04:20.406633] [disk (free/total): 18.9GB/25.0GB]
Gate task times:
  0:04:20.405428    CTW:hosted
  =======
  0:04:20.406633
CompileTheWorld : Done (28662 classes, 201631 methods, 259438739407 ms elapsed, 7438605 ms compile time, 141075341888 bytes of memory used)
gate: 05 Nov 2019 16:09:47(+04:21) END:   CTW:hosted [0:04:21.980836] [disk (free/total): 18.9GB/25.0GB]
WARNING: Removing scaladacapo:actors from benchmarks because corba has been removed since JDK11 (http://openjdk.java.net/jeps/320)
gate: 05 Nov 2019 16:09:47(+04:21) END:   Gate [0:04:21.981966] [disk (free/total): 18.9GB/25.0GB]
Gate task times:
  0:04:21.980836    CTW:hosted
  =======
  0:04:21.981966
sanzinger picture sanzinger on 5 Nov 2019

Given that it's a SIGILL I'd be highly suspicious of Graal emitting an instruction that is not supported on the target chip. This would explain why it appears on one system and not another. Might be worth comparing the Features lines from /proc/cpuinfo on the two systems (the one that works and the one that doesn't work).

dmlloyd picture dmlloyd on 5 Nov 2019
👍1

OTOH the pc value doesn't look so good.

dmlloyd picture dmlloyd on 5 Nov 2019

Hi @sanzinger , this issue doesn't always happen but with a high probability. This is the stack trace from the hs_err_pid file:

---------------  T H R E A D  ---------------

Current thread (0x0000ffff84014000):  JavaThread "main" [_thread_in_Java, id=24332, 
stack(0x0000ffff890ce000,0x0000ffff892ce000)]

Stack: [0x0000ffff890ce000,0x0000ffff892ce000],  sp=0x0000ffff892cada0,  free space=2035k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0000000800001ac8
j  org.graalvm.compiler.code.CompilationResult.close()V+38 jdk.internal.vm.compiler
j  org.graalvm.compiler.lir.asm.CompilationResultBuilder.closeCompilationResult()V+4 jdk.internal.vm.compiler
j  org.graalvm.compiler.lir.asm.CompilationResultBuilder.finish()V+89 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.gen.LIRCompilerBackend.emitCode(Lorg/graalvm/compiler/core/target/Backend;Ljdk/vm/ci/meta/Assumptions;Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljava/util/Collection;Lorg/graalvm/collections/EconomicSet;Ljdk/vm/ci/meta/SpeculationLog;ILorg/graalvm/compiler/lir/gen/LIRGenerationResult;Lorg/graalvm/compiler/code/CompilationResult;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;)V+127 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.gen.LIRCompilerBackend.emitBackEnd(Lorg/graalvm/compiler/nodes/StructuredGraph;Ljava/lang/Object;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/core/target/Backend;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Ljdk/vm/ci/code/RegisterConfig;Lorg/graalvm/compiler/lir/phases/LIRSuites;)V+118 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.target.Backend.emitBackEnd(Lorg/graalvm/compiler/nodes/StructuredGraph;Ljava/lang/Object;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Ljdk/vm/ci/code/RegisterConfig;Lorg/graalvm/compiler/lir/phases/LIRSuites;)V+12 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.stubs.Stub.buildCompilationResult(Lorg/graalvm/compiler/debug/DebugContext;Lorg/graalvm/compiler/core/target/Backend;)Lorg/graalvm/compiler/code/CompilationResult;+200 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.stubs.Stub.getCode(Lorg/graalvm/compiler/core/target/Backend;)Ljdk/vm/ci/code/InstalledCode;+45 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.HotSpotForeignCallLinkageImpl.finalizeAddress(Lorg/graalvm/compiler/core/target/Backend;)V+35 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.meta.HotSpotForeignCallsProviderImpl.lookupForeignCall(Lorg/graalvm/compiler/core/common/spi/ForeignCallDescriptor;)Lorg/graalvm/compiler/hotspot/HotSpotForeignCallLinkage;+73 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.meta.HotSpotForeignCallsProviderImpl.lookupForeignCall(Lorg/graalvm/compiler/core/common/spi/ForeignCallDescriptor;)Lorg/graalvm/compiler/core/common/spi/ForeignCallLinkage;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.aarch64.AArch64HotSpotBackend.emitCodeSuffix(Lorg/graalvm/compiler/lir/asm/CompilationResultBuilder;Lorg/graalvm/compiler/asm/aarch64/AArch64MacroAssembler;Lorg/graalvm/compiler/lir/framemap/FrameMap;)V+63 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.aarch64.AArch64HotSpotBackend.emitCode(Lorg/graalvm/compiler/lir/asm/CompilationResultBuilder;Lorg/graalvm/compiler/lir/LIR;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/asm/Label;)V+47 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.aarch64.AArch64HotSpotBackend.emitCode(Lorg/graalvm/compiler/lir/asm/CompilationResultBuilder;Lorg/graalvm/compiler/lir/LIR;Ljdk/vm/ci/meta/ResolvedJavaMethod;)V+20 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.gen.LIRCompilerBackend.emitCode(Lorg/graalvm/compiler/core/target/Backend;Ljdk/vm/ci/meta/Assumptions;Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljava/util/Collection;Lorg/graalvm/collections/EconomicSet;Ljdk/vm/ci/meta/SpeculationLog;ILorg/graalvm/compiler/lir/gen/LIRGenerationResult;Lorg/graalvm/compiler/code/CompilationResult;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;)V+63 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.gen.LIRCompilerBackend.emitBackEnd(Lorg/graalvm/compiler/nodes/StructuredGraph;Ljava/lang/Object;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/core/target/Backend;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Ljdk/vm/ci/code/RegisterConfig;Lorg/graalvm/compiler/lir/phases/LIRSuites;)V+118 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.target.Backend.emitBackEnd(Lorg/graalvm/compiler/nodes/StructuredGraph;Ljava/lang/Object;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Ljdk/vm/ci/code/RegisterConfig;Lorg/graalvm/compiler/lir/phases/LIRSuites;)V+12 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.GraalCompiler.compile(Lorg/graalvm/compiler/core/GraalCompiler$Request;)Lorg/graalvm/compiler/code/CompilationResult;+129 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.GraalCompiler.compileGraph(Lorg/graalvm/compiler/nodes/StructuredGraph;Ljdk/vm/ci/meta/ResolvedJavaMethod;Lorg/graalvm/compiler/phases/util/Providers;Lorg/graalvm/compiler/core/target/Backend;Lorg/graalvm/compiler/phases/PhaseSuite;Lorg/graalvm/compiler/phases/OptimisticOptimizations;Ljdk/vm/ci/meta/ProfilingInfo;Lorg/graalvm/compiler/phases/tiers/Suites;Lorg/graalvm/compiler/lir/phases/LIRSuites;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Z)Lorg/graalvm/compiler/code/CompilationResult;+27 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileHelper(Lorg/graalvm/compiler/lir/asm/CompilationResultBuilderFactory;Lorg/graalvm/compiler/code/CompilationResult;Lorg/graalvm/compiler/nodes/StructuredGraph;Ljdk/vm/ci/meta/ResolvedJavaMethod;IZZLorg/graalvm/compiler/options/OptionValues;)Lorg/graalvm/compiler/code/CompilationResult;+215 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compile(Ljdk/vm/ci/meta/ResolvedJavaMethod;IZZLorg/graalvm/compiler/core/common/CompilationIdentifier;Lorg/graalvm/compiler/debug/DebugContext;)Lorg/graalvm/compiler/code/CompilationResult;+47 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+113 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.CompilationWrapper.run(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+114 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation(Lorg/graalvm/compiler/options/OptionValues;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+44 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.compile(Ljava/lang/String;Lorg/graalvm/compiler/hotspot/test/CompileTheWorld$LibGraalParams;)V+102
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.compile()V+152
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.main([Ljava/lang/String;)V+109
v  ~StubRoutines::call_stub
V  [libjvm.so+0x69682c]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x354
V  [libjvm.so+0x707c20]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.75] [clone .constprop.159]+0x130
V  [libjvm.so+0x709ad0]  jni_CallStaticVoidMethod+0xe0
C  [libjli.so+0x569c]  JavaMain+0xe84
C  [libjli.so+0x8a50]  call_continuation+0x10
C  [libpthread.so.0+0x7088]  start_thread+0xb0

siginfo: si_signo: 4 (SIGILL), si_code: 1 (ILL_ILLOPC), si_addr: 0x0000000800001ac8
XiaohongGong picture XiaohongGong on 7 Nov 2019

Might be worth comparing the Features lines from /proc/cpuinfo on the two systems (the one that works and the one that doesn't work).

@dmlloyd make sense! This is my CPU Features from /proc/cpuinfo :

Features : fp asimd aes pmull sha1 sha2 crc32

XiaohongGong picture XiaohongGong on 7 Nov 2019

More information to 0x0000000800001ac8:

Registers:
R0 =0x0000000000000300 is an unknown value
R1 =0x00000000ffc35110 points into unknown readable memory: 01 00 00 00 00 00 00 00
R2 =0x0 is NULL
R3 =0x0 is NULL
R4 =0x00000000feab4960 is an oop: org.graalvm.compiler.code.DataSection$$Lambda$54/0x0000000840230440
{0x00000000feab4960} - klass: 'org/graalvm/compiler/code/DataSection$$Lambda$54'
...
R8 =0x0000000800001ac8 is pointing into metadata

Instructions: (pc=0x0000000800001ac8)
0x0000000800001ac8:   08 1e 9e d2 c8 11 ad f2 e8 ff df f2 00 01 1f d6
XiaohongGong picture XiaohongGong on 7 Nov 2019

Hi, it's a hotspot issue. I created a JBS here: https://bugs.openjdk.java.net/browse/JDK-8234321

XiaohongGong picture XiaohongGong on 18 Nov 2019

Patch merged in jdk master: https://hg.openjdk.java.net/jdk/jdk/rev/2c55c2fc08f5
And it will also be back ported to jdk11u. Thanks!

XiaohongGong picture XiaohongGong on 21 Nov 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

guaporocco picture guaporocco  路  3Comments
oroppas picture oroppas  路  3Comments
jinwuxia picture jinwuxia  路  3Comments
borkdude picture borkdude  路  3Comments
schneidersteve picture schneidersteve  路  3Comments