Openj9: HCRLateAttachWorkload system test fails (OpenJDK8U-jdk_x64_linux_openj9_2018-12-05-19-28)

Created on 6 Dec 2018 · 16Comments · Source: eclipse/openj9

When running grinders to re-enable this test, it fails/crashes when run against recent nightly build at openj9.

Snippet from https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/514/console:

04:09:44 LT stderr Unhandled exception
04:09:44 LT stderr Type=Segmentation error vmState=0x00000000
04:09:44 LT stderr J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000080
04:09:44 LT stderr Handler1=00007FE9E11D89B0 Handler2=00007FE9E0AD6440 InaccessibleAddress=0000000000000000
04:09:44 LT stderr RDI=00007FE9A80012F8 RSI=0000000000000000 RAX=00007FE9D8A71A30 RBX=6EC5459800000000
04:09:44 LT stderr RCX=00007FE96C002F60 RDX=00007FE96C002F40 R8=0000000000000000 R9=00007FE9D91AA2B0
04:09:44 LT stderr R10=0000000000000001 R11=0000000000000283 R12=00000000FFF6E500 R13=00000000013C7600
04:09:44 LT stderr R14=00000000FFE60B38 R15=0000000000000000
04:09:44 LT stderr RIP=00007FE9E1154DF8 GS=0000 FS=0000 RSP=00007FE9D8A716A0
04:09:44 LT stderr EFlags=0000000000010206 CS=0033 RBP=0000000001411630 ERR=0000000000000000
04:09:44 LT stderr TRAPNO=000000000000000D OLDMASK=0000000000000000 CR2=00007F7539D62E00
04:09:44 LT stderr xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:09:44 LT stderr xmm1 00000000013d9a1a (f: 20814362.000000, d: 1.028366e-316)
04:09:44 LT stderr xmm2 00007fe9d8a71a60 (f: 3634829824.000000, d: 6.948655e-310)
04:09:44 LT stderr xmm3 00000000fff6c430 (f: 4294362112.000000, d: 2.121697e-314)
04:09:44 LT stderr xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
04:09:44 LT stderr xmm5 00000000e01e3d60 (f: 3760078080.000000, d: 1.857725e-314)
04:09:44 LT stderr xmm6 00000000ffb47b70 (f: 4290018048.000000, d: 2.119551e-314)
04:09:44 LT stderr xmm7 00000000e01b2fb0 (f: 3759878144.000000, d: 1.857627e-314)
04:09:44 LT stderr xmm8 3bbcc86800000000 (f: 0.000000, d: 6.095003e-21)
04:09:44 LT stderr xmm9 3f81e7d73bb2fb65 (f: 1001585536.000000, d: 8.742982e-03)
04:09:44 LT stderr xmm10 3d75461651d7a03d (f: 1373085696.000000, d: 1.209274e-12)
04:09:44 LT stderr xmm11 402e84161466f9bf (f: 342292928.000000, d: 1.525798e+01)
04:09:44 LT stderr xmm12 bcb3600000000000 (f: 0.000000, d: -2.688821e-16)
04:09:44 LT stderr xmm13 bf81e7d73bb2fc00 (f: 1001585664.000000, d: -8.742982e-03)
04:09:44 LT stderr xmm14 402e7f9c1e980d00 (f: 513281280.000000, d: 1.524924e+01)
04:09:44 LT stderr xmm15 bbce0bb59d421e6a (f: 2638355968.000000, d: -1.272487e-20)
04:09:44 LT stderr Module=/home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
04:09:44 LT stderr Module_base_address=00007FE9E113B000
04:09:44 LT stderr Target=2_90_20181205_130 (Linux 4.4.0-116-generic)
04:09:44 LT stderr CPU=amd64 (4 logical CPUs) (0x1f227e000 RAM)
04:09:44 LT stderr ----------- Stack Backtrace -----------
04:09:44 LT stderr (0x00007FE9E1154DF8 [libj9vm29.so+0x19df8])
04:09:44 LT stderr (0x00007FE9E1153D20 [libj9vm29.so+0x18d20])
04:09:44 LT stderr (0x00007FE9E1212DD2 [libj9vm29.so+0xd7dd2])
04:09:44 LT stderr ---------------------------------------
04:09:44 LT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2018/12/06 03:09:43 - please wait.
04:09:44 LT stderr JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Grinder/openjdk-tests/TestConfig/test_output_15440657597339/HCRLateAttachWorkload_0/20181206-030920-HCRLateAttachWorkload/results/core.20181206.030943.29201.0001.dmp' in response to an event

Test passed when run against hotspot build:
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/513/console

Note: for jdk11 run, test still fails with known test issue https://github.com/adoptopenjdk/openjdk-systemtest/issues/7.

test failure

Source

smlambert

All 16 comments

Core and jitdump temporarily housed here

smlambert on 6 Dec 2018

@DanHeidinga

pshipton on 6 Dec 2018

@gacholio Can you take a look at this?

DanHeidinga on 6 Dec 2018

We should also check the JVM used to test has the fixes
https://github.com/eclipse/openj9/pull/3919 - commit sha dfb5c08, merge sha 8c4e124
https://github.com/eclipse/omr/pull/3282 - commit sha 04c5a3d

04:04:22 OpenJDK Runtime Environment (build 1.8.0_192-201812050430-b12)
04:04:22 Eclipse OpenJ9 VM (build master-cdc67435b, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181205_130 (JIT enabled, AOT enabled)
04:04:22 OpenJ9   - cdc67435b
04:04:22 OMR      - 3a90b31b
04:04:22 JCL      - 590cbb791c based on jdk8u192-b12)

git merge-base --is-ancestor dfb5c08 cdc67435b && echo yes seems to be the easy way to determine this.

pshipton on 6 Dec 2018

@smlambert doesn't look like the JVM used has the required fixes. Can you please try it again on a later build. The fix was merged yesterday.

pshipton on 6 Dec 2018

👍1

The core above won't load in DDR, so I'm going to wait until the re-test before spending any more time.

gacholio on 6 Dec 2018

👍1

Passes against last nights build, https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/525/console, will re-enable and close this issue.

smlambert on 6 Dec 2018

Re-enabled on jdk8: https://github.com/AdoptOpenJDK/openjdk-tests/pull/730

smlambert on 6 Dec 2018

🎉1

@gacholio I'm concerned about "The core above won't load in DDR" as this is supposed to work.
a) how did it fail to load?
b) did you use OpenJ9 to open the core?
c) does the core seem truncated or some other known problem?

pshipton on 6 Dec 2018

@gacholio I was able to open the core with DDR, I suspect you didn't use an OpenJ9 JVM to open it.

pshipton on 6 Dec 2018

I did, but it was an old one. Also, that's not supposed to matter.

The error I got was "no jre".

gacholio on 6 Dec 2018

Also, that's not supposed to matter.

There were (and are) still some outstanding issues to work through with DDR. If it doesn't work probably best to try the matching JVM until the issues are all worked out. I believe there will be at least one more breakage coming which affects reading older core files.

pshipton on 6 Dec 2018

Also, no guarantee an older JVM can read a newer core file. Although the latest JVM should be able to read all older ones (but I expect this won't be true for OpenJ9 until all the issues are resolved).

pshipton on 6 Dec 2018

I ran the HCRLateAttachWorkload test on jdk11 OpenJ9 Linux SDK and received a different error. I wonder if it is related to the out dated ASM issue?

Below are the details (Please let me know if you want a new issue raised for this).

16:59:20  LT  stderr Exception in thread "Attachment 53082" java.lang.reflect.InvocationTargetException
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:59:20  LT  stderr    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibraryImpl(Native Method)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibrary(Attachment.java:328)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.parseLoadAgent(Attachment.java:306)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.doCommand(Attachment.java:183)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.run(Attachment.java:143)
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.AgentLogger.throwErrors(AgentLogger.java:52)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringModifierAgent.agentmain(StringModifierAgent.java:131)
16:59:20  LT  stderr    ... 11 more
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:84)
16:59:20  LT  stderr    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
16:59:20  LT  stderr    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.TransformerMakerThread.run(TransformerMakerThread.java:68)
16:59:20  LT  stderr Caused by: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:70)
16:59:20  LT  stderr    ... 6 more
16:59:20  LT  stderr Agent failed to start!
16:59:20  LT  stderr JVMJ9TI064E Agent initialization function Agent_OnAttach failed for library instrument, return code 102
16:59:20  AG  stderr Exception in thread "main" com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException102
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.parseResponse(OpenJ9VirtualMachine.java:321)
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.loadAgent(OpenJ9VirtualMachine.java:229)
16:59:20  AG  stderr    at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:75)
16:59:20  LT  stderr Exception in thread "Attachment 53082" java.lang.reflect.InvocationTargetException
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:59:20  LT  stderr    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibraryImpl(Native Method)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibrary(Attachment.java:328)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.parseLoadAgent(Attachment.java:306)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.doCommand(Attachment.java:183)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.run(Attachment.java:143)
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.AgentLogger.throwErrors(AgentLogger.java:52)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringModifierAgent.agentmain(StringModifierAgent.java:131)
16:59:20  LT  stderr    ... 11 more
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:84)
16:59:20  LT  stderr    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
16:59:20  LT  stderr    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.TransformerMakerThread.run(TransformerMakerThread.java:68)
16:59:20  LT  stderr Caused by: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:70)
16:59:20  LT  stderr    ... 6 more
16:59:20  LT  stderr Agent failed to start!
16:59:20  LT  stderr JVMJ9TI064E Agent initialization function Agent_OnAttach failed for library instrument, return code 102
16:59:20  AG  stderr Exception in thread "main" com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException102
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.parseResponse(OpenJ9VirtualMachine.java:321)
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.loadAgent(OpenJ9VirtualMachine.java:229)
16:59:20  AG  stderr    at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:75)v16:59:20  LT  stderr Exception in thread "Attachment 53082" java.lang.reflect.InvocationTargetException
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:59:20  LT  stderr    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:59:20  LT  stderr    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibraryImpl(Native Method)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.loadAgentLibrary(Attachment.java:328)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.parseLoadAgent(Attachment.java:306)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.doCommand(Attachment.java:183)
16:59:20  LT  stderr    at java.base/com.ibm.tools.attach.target.Attachment.run(Attachment.java:143)
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.AgentLogger.throwErrors(AgentLogger.java:52)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringModifierAgent.agentmain(StringModifierAgent.java:131)
16:59:20  LT  stderr    ... 11 more
16:59:20  LT  stderr Caused by: java.lang.Error: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:84)
16:59:20  LT  stderr    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
16:59:20  LT  stderr    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
16:59:20  LT  stderr    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.TransformerMakerThread.run(TransformerMakerThread.java:68)
16:59:20  LT  stderr Caused by: java.lang.IllegalArgumentException
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at org.objectweb.asm.ClassReader.<init>(Unknown Source)
16:59:20  LT  stderr    at net.adoptopenjdk.test.hcrAgent.agent.StringTransformer.transform(StringTransformer.java:70)
16:59:20  LT  stderr    ... 6 more
16:59:20  LT  stderr Agent failed to start!
16:59:20  LT  stderr JVMJ9TI064E Agent initialization function Agent_OnAttach failed for library instrument, return code 102
16:59:20  AG  stderr Exception in thread "main" com.sun.tools.attach.AgentInitializationException: ATTACH_ERR AgentInitializationException102
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.parseResponse(OpenJ9VirtualMachine.java:321)
16:59:20  AG  stderr    at jdk.attach/com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.loadAgent(OpenJ9VirtualMachine.java:229)
16:59:20  AG  stderr    at net.adoptopenjdk.test.hcrAgent.agent.Attacher.main(Attacher.java:75)

Java version:

openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+2-201903181910)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-f6ba2734e, JRE 11 Linux amd64-64-Bit Compressed References 20190318_172 (JIT enabled, AOT enabled)
OpenJ9   - f6ba2734e
OMR      - 9b73e2bd
JCL      - 3cd1a589af based on jdk-11.0.3+2)

Grinder link:
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1327/console