Openj9: extended.functional testvmcheck_6 crash vmState=0x0005ffff

Created on 26 Sep 2019  路  44Comments  路  Source: eclipse/openj9

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/153/

===============================================
Running test testvmcheck_6 ...
===============================================
testvmcheck_6 Start Time: Thu Sep 26 00:28:11 2019 Epoch Time (ms): 1569472091218
variation: Mode551
JVM_OPTIONS: -Xcompressedrefs -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced 
[IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TestConfig/scripts/testKitGen/../../../../jvmtest/TestConfig/resources/excludes/latest_exclude_8.txt
...
... TestNG 6.14.2 by C茅dric Beust ([email protected])
...

[TestLoadingClassesFromJarfile] [WARN] /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/jre/lib/compressedrefs/jclSC180/vm.jar does not exist
[TestLoadingClassesFromJarfile] [WARN] /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/jre/lib/se-service.jar does not exist
FAILED: testLoadingClasses
java.lang.AssertionError: Child exited with non-zero stats expected:<0> but was:<255>
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
    at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
    at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
    at org.openj9.test.vmcheck.TestLoadingClassesFromJarfile.createAndRunNewClassloadingTester(TestLoadingClassesFromJarfile.java:203)
    at org.openj9.test.vmcheck.TestLoadingClassesFromJarfile.loadClassesFromJarfile(TestLoadingClassesFromJarfile.java:115)
    at org.openj9.test.vmcheck.TestLoadingClassesFromJarfile.testLoadingClasses(TestLoadingClassesFromJarfile.java:80)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:648)
    at org.testng.TestRunner.run(TestRunner.java:505)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
    at org.testng.SuiteRunner.run(SuiteRunner.java:364)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
    at org.testng.TestNG.runSuites(TestNG.java:1049)
    at org.testng.TestNG.run(TestNG.java:1017)
    at org.testng.TestNG.privateMain(TestNG.java:1354)
    at org.testng.TestNG.main(TestNG.java:1323)

1XHEXCPMODULE  Module: /usr/lib/system/libsystem_c.dylib
1XHEXCPMODULE  Module_base_address: 00007FFF83189000
1XHEXCPMODULE  Symbol: __findenv
1XHEXCPMODULE  Symbol_address: 00007FFF831E8780
NULL           
1XHREGISTERS   Registers:
2XHREGISTER      RDI: 000000001076D36B
2XHREGISTER      RSI: 00007000002DC394
2XHREGISTER      RAX: 0000000000000000
2XHREGISTER      RBX: 000000001076D36B
2XHREGISTER      RCX: 000000000000004A
2XHREGISTER      RDX: 00007FC7B37295B0
2XHREGISTER      R8: 00000000FFFFFFF3
2XHREGISTER      R9: 000000000000000D
2XHREGISTER      R10: 00007FC7B3729828
2XHREGISTER      R11: 000800000E950A30
2XHREGISTER      R12: 00000000FFFFFFF3
2XHREGISTER      R13: 000000002E597740
2XHREGISTER      R14: 000000000000004A
2XHREGISTER      R15: 0000000000000054
2XHREGISTER      XMM0: 0000000000000000
2XHREGISTER      XMM1: 000000057FFFFFFF
2XHREGISTER      XMM2: 000000001074D3DD
2XHREGISTER      XMM3: 0000000000000000
2XHREGISTER      XMM4: 0000009600000320
2XHREGISTER      XMM5: 0000000000000000
2XHREGISTER      XMM6: 0000000000000000
2XHREGISTER      XMM7: 0000000000000000
2XHREGISTER      XMM8: 0000000000000000
2XHREGISTER      XMM9: 0000000000000000
2XHREGISTER      XMM10: 0000000000000000
2XHREGISTER      XMM11: 0000000000000000
2XHREGISTER      XMM12: 0000000000000000
2XHREGISTER      XMM13: 0000000000000000
2XHREGISTER      XMM14: 0000000000000000
2XHREGISTER      XMM15: 0000000000000000
2XHREGISTER      RIP: 00007FFF831E87D4
2XHREGISTER      GS: 0000
2XHREGISTER      FS: 0000
2XHREGISTER      RSP: 00007000002DC360
2XHREGISTER      RFlags: 0000000000010202
2XHREGISTER      CS: 002B
2XHREGISTER      RBP: 00007000002DC380
2XHREGISTER      ERR: FFF7700000000000
2XHREGISTER      TRAPNO: 000000000000000D
2XHREGISTER      CPU: 7000000000000000
2XHREGISTER      FAULTVADDR: 00000000FFF77000
1XHEXCPMODULE  Compiling method: java/lang/Integer.toUnsignedLong(I)J
NULL           
1XHFLAGS       VM flags:000000000005FFFF



md5-562971688ccd78fc399bd7a86cca726d



00:38:07  To rebuild the failed test in a jenkins job, copy the following link and fill out the <Jenkins URL> and <FAILED test target>:
00:38:07  <Jenkins URL>/parambuild/?JDK_VERSION=8&JDK_IMPL=openj9&BUILD_LIST=functional&JenkinsFile=openjdk_x86-64_mac&TARGET=<FAILED test target>
00:38:07  
00:38:07  For example, to rebuild the failed tests in <Jenkins URL>=https://ci.adoptopenjdk.net/job/Grinder, use the following links:
00:38:07  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=8&JDK_IMPL=openj9&BUILD_LIST=functional&JenkinsFile=openjdk_x86-64_mac&TARGET=testvmcheck_6
jit segfault test failure
Source
picture pshipton

All 44 comments

@andrewcraik

pshipton picture pshipton on 26 Sep 2019

https://ci.eclipse.org/openj9/job/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/62
J9vmTest_5

+++ j9vm.test.threadname.ASCIIThreadNameTest: +++
command: /Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdkbinary/j2sdk-image/bin/java -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump -Xms64m -Xmx64m    -classpath /Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdk-tests/TestConfig/../../jvmtest/functional/VM_Test/VM_Test.jar:/Users/jenkins/workspace/Test_openjdknext_j9_sanity.functional_x86-64_mac_OpenJDK/openjdk-tests/TestConfig/../../jvmtest/TestConfig/lib/asm-all.jar  j9vm.test.threadname.ASCIIThreadNameTest 

Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000749B9A0 Handler2=0000000007693880
RDI=0000000008A0CB71 RSI=0000700000259384 RAX=0000000000000000 RBX=0000000008A0CB71
RCX=000000000000004A RDX=00007FD51143DC40 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FD51143DEC8 R11=004F000000000000 R12=00000000FFFFFFF3 R13=000000001D5BA4B0
R14=000000000000004A R15=0000000000000054
RIP=00007FFF9E9BB7D4 GS=0000 FS=0000 RSP=0000700000259350
RFlags=0000000000010202 CS=002B RBP=0000700000259370 ERR=41F7E00000000000
TRAPNO=000000000000000D CPU=E000000000000000 FAULTVADDR=0000000041F7E000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
XMM3 6e616c2f6176616a (f: 1635148160.000000, d: 5.038250e+223)
XMM4 616c432f676e616c (f: 1735287168.000000, d: 1.986723e+161)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF9E95C000 Symbol=__findenv
Symbol_address=00007FFF9E9BB780

Method_being_compiled=java/lang/ClassLoader.isVerboseImpl()Z
Target=2_90_20191027_79 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton picture pshipton on 28 Oct 2019 
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF9E95C000 Symbol=__findenv

This is the findenv crash again. Looking through OpenJ9, the only setenv I can find is:
https://github.com/eclipse/openj9/blob/fc509ea10664ba42c94498030a69a1b36afc5118/runtime/j9vm/jvm.c#L2487

Looking through OMR, I only see references in AUXV:
https://github.com/eclipse/omr/blob/f8fc704032f95cbc0f337eb6be54f2b06e58a773/port/unix_include/auxv.h#L23-L25

Looking in the Extensions repo, the references I see are either in the launcher code or AWT related. I wouldn't expect either to changing after startup.

DanHeidinga picture DanHeidinga on 28 Oct 2019

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly/198
osx1011-x86-2
testSCCMLTests6_1

===============================================
Running test testSCCMLTests6_1 ...
===============================================
testSCCMLTests6_1 Start Time: Mon Nov 25 02:46:51 2019 Epoch Time (ms): 1574668011600
variation: Mode610
JVM_OPTIONS: -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon 
Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=0000000001294540 Handler2=00000000014A37F0
RDI=0000000003035192 RSI=0000700000259374 RAX=0000000000000000 RBX=0000000003035192
RCX=000000000000004F RDX=00007FBE72F3C5D0 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FBE72F3C858 R11=A00007FBE72F3C15 R12=00000000FFFFFFF3 R13=000000002C21FCD0
R14=000000000000004F R15=0000000000000054
RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=0000700000259340
RFlags=0000000000010202 CS=002B RBP=0000700000259360 ERR=C1471FA800000000
TRAPNO=000000000000000D CPU=1FA8000000000000 FAULTVADDR=00000000C1471FA8
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF809D5000 Symbol=__findenv
Symbol_address=00007FFF80A34780

Method_being_compiled=java/util/zip/Inflater.reset(J)V
Target=2_90_20191124_207 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton picture pshipton on 25 Nov 2019

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Release/19/
cmdLineTester_classesdbgddrext_1
Run !dumpallramclasslinear 10

 [OUT] DTFJView version 4.29.5, using DTFJ version 1.12.29003
 [OUT] Loading image from DTFJ...
 [OUT] 
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
 [ERR] Handler1=00000000082A1ED0 Handler2=0000000007937950
 [ERR] RDI=0000000009704B6F RSI=0000700000259354 RAX=0000000000000000 RBX=0000000009704B6F
 [ERR] RCX=000000000000004A RDX=00007F9849400770 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007F9849400A18 R11=002B000000000000 R12=00000000FFFFFFF3 R13=00007000002599E8
 [ERR] R14=000000000000004A R15=0000000000000054
 [ERR] RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=0000700000259320
 [ERR] RFlags=0000000000010202 CS=002B RBP=0000700000259340 ERR=3BE6100000000000
 [ERR] TRAPNO=000000000000000D CPU=1000000000000000 FAULTVADDR=000000003BE61000
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 000000057fffffff (f: 2147483648.000000, d: 1.167098e-313)
 [ERR] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF809D5000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF80A34780
 [ERR] 
 [ERR] Method_being_compiled=java/util/zip/ZipFile.read(JJJ[BII)I
 [ERR] Target=2_90_20200104_21 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton picture pshipton on 4 Jan 2020

https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_extended.functional_x86-64_mac_Release/12
testSCCacheManagement_0

.Running  :testCreationAndCompatibility  (Test suite : TestOptionsCacheDir)
SimpleApp running

[-Xshareclasses persistent cache enabled]
[-Xshareclasses verbose output enabled]
JVMSHRC236I Created shared classes persistent cache Foo
JVMSHRC246I Attached shared classes persistent cache Foo
JVMSHRC765I Memory page protection on runtime data, string read-write data and partially filled pages is successfully enabled
Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000E09B540 Handler2=000000000D6FF5C0
RDI=000000000F4EB277 RSI=0000700000259384 RAX=0000000000000000 RBX=000000000F4EB277
RCX=000000000000004A RDX=00007FE498529B20 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FE498529DE0 R11=20AB8D9ABADBAD21 R12=00000000FFFFFFF3 R13=0000000028D1E390
R14=000000000000004A R15=0000000000000054
RIP=00007FFF9C7777D4 GS=0000 FS=0000 RSP=0000700000259350
RFlags=0000000000010202 CS=002B RBP=0000700000259370 ERR=9870102800000000
TRAPNO=000000000000000D CPU=1028000000000000 FAULTVADDR=00007FE498701028
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 000000057fffffff (f: 2147483648.000000, d: 1.167098e-313)
XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF9C718000 Symbol=__findenv
Symbol_address=00007FFF9C777780

Method_being_compiled=java/lang/Thread.startImpl()V
Target=2_90_20200104_13 (Mac OS X 10.11.6)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
pshipton picture pshipton on 5 Jan 2020

From an internal build:

21:47:54  openjdk version "14-internal" 2020-03-17
21:47:54  OpenJDK Runtime Environment (build 14-internal+0-adhoc.jenkins.BuildJDK14x86-64macNightly)
21:47:54  Eclipse OpenJ9 VM (build ibm_sdk-479247ea9a, JRE 14 Mac OS X amd64-64-Bit Compressed References 20200204_18 (JIT enabled, AOT enabled)
21:47:54  OpenJ9   - 479247ea9a
21:47:54  OMR      - b6a8e9309
21:47:54  JCL      - 9820fbd5aea based on jdk-14+34)

00:20:02  Testing: -verbose:gc -verbose 
00:20:02  Test start time: 2020/02/05 00:20:01 Eastern Standard Time
00:20:02  Running command: "/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xcompressedrefs -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump -verbose:gc -verbose  -cp "/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../../jvmtest/TestConfig/resources:/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../TKG/lib/testng.jar:/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../TKG/lib/jcommander.jar:/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../../jvmtest/functional/JIT_Test/jitt.jar" org.testng.TestNG -d "/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../TKG/test_output_15808715252771/cmdLineTester_verbosetest_3" "/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/../../jvmtest/functional/JIT_Test/testng.xml" -testnames AllocationTest -groups level.extended
00:20:02  Time spent starting: 4 milliseconds
00:20:54  Time spent executing: 46328 milliseconds
00:20:54  Test result: FAILED

[ERR] class load: java/lang/Class$ClassReflectNullPlaceHolder from: jrt:/java.base 
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [ERR] Handler1=000000000869B560 Handler2=0000000008883530 InaccessibleAddress=0000000000000150
 [ERR] RDI=0000000009C0C497 RSI=00000000C0258374 RAX=0000000000000000 RBX=0000000009C0C497
 [ERR] RCX=000000000000004A RDX=00007F924944BA90 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007F924944BD98 R11=0000000000000150 R12=00000000210AF5A0 R13=0000000021036080
 [ERR] R14=00000000FFFFFFF3 R15=0000000000000054
 [ERR] RIP=00007FFF909ACAA3 GS=0000 FS=0000 RSP=00000000C0258348
 [ERR] RFlags=0000000000010202 CS=002B RBP=00000000C0258360 ERR=0000015000000004
 [ERR] TRAPNO=000000040000000E CPU=0150000000040000 FAULTVADDR=0000000000000150
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
 [ERR] XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
 [ERR] XMM3 2f6f692f6176616a (f: 1635148160.000000, d: 3.311397e-80)
 [ERR] XMM4 63416d6f646e6152 (f: 1684955520.000000, d: 1.315413e+170)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF9094E000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF909ACA49
 [ERR] 
 [ERR] Method_being_compiled=java/io/RandomAccessFile.length()J
 [ERR] Target=2_90_20200204_18 (Mac OS X 10.10.5)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
 [ERR] JVMDUMP039I Processing dump event "gpf", detail "" at 2020/02/05 00:20:03 - please wait.
 [ERR] class load: org.testng.log4testng.Logger from: file:/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/lib/testng.jar
 [ERR] class load: java/io/ByteArrayOutputStream from: jrt:/java.base 
 [ERR] JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/core.20200205.002003.786.0001.dmp' in response to an event
 [ERR] JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/core.20200205.002003.786.0001.dmp
 [ERR] JVMDUMP032I JVM requested Java dump using '/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/javacore.20200205.002003.786.0002.txt' in response to an event
 [ERR] JVMDUMP010I Java dump written to /Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/javacore.20200205.002003.786.0002.txt
 [ERR] JVMDUMP032I JVM requested Snap dump using '/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/Snap.20200205.002003.786.0003.trc' in response to an event
 [ERR] JVMDUMP010I Snap dump written to /Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/Snap.20200205.002003.786.0003.trc
 [ERR] JVMDUMP007I JVM Requesting JIT dump using '/Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/jitdump.20200205.002003.786.0004.dmp'
 [ERR] JVMDUMP010I JIT dump written to /Users/jenkins/workspace/Test_openjdk14_j9_extended.functional_x86-64_mac_Nightly/openjdk-tests/TKG/test_output_15808715252771/cmdLineTester_verbosetest_3/jitdump.20200205.002003.786.0004.dmp
 [ERR] JVMDUMP013I Processed dump event "gpf", detail "".
>> Required condition was found: [Output match: <verbosegc]
>> Required condition was found: [Output match: class load: java/lang/Object]
>> Failure condition was not found: [Output match: <JNI FindClass: java/lang/Class>]
>> Success condition was not found: [Output match: 100000 x ObjectAllocation.allocArrays]

To rebuild the failed tests in =https://ci.adoptopenjdk.net/job/Grinder, use the following links:
01:46:25 https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=14&JDK_IMPL=openj9&BUILD_LIST=functional&PLATFORM=x86-64_mac&TARGET=cmdLineTester_verbosetest_3

JasonFengJ9 picture JasonFengJ9 on 6 Feb 2020

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly/279
building the test material

23:09:28      [javac] Unhandled exception
23:09:28      [javac] Type=Segmentation error vmState=0x0005ffff
23:09:28      [javac] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
23:09:28      [javac] Handler1=000000001049B1D0 Handler2=000000000FAED370
23:09:28      [javac] RDI=00000000118F91C2 RSI=0000700000259384 RAX=0000000000000000 RBX=00000000118F91C2
23:09:28      [javac] RCX=000000000000004A RDX=00007FDC83602300 R8=00000000FFFFFFF3 R9=000000000000000D
23:09:28      [javac] R10=00007FDC83602598 R11=0109000000006B63 R12=00000000FFFFFFF3 R13=0000000028A1D1D0
23:09:28      [javac] R14=000000000000004A R15=0000000000000054
23:09:28      [javac] RIP=00007FFF80A347D4 GS=0000 FS=0000 RSP=0000700000259350
23:09:28      [javac] RFlags=0000000000010202 CS=002B RBP=0000700000259370 ERR=8386822800000000
23:09:28      [javac] TRAPNO=000000000000000D CPU=8228000000000000 FAULTVADDR=00007FDC83868228
23:09:28      [javac] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
23:09:28      [javac] XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
23:09:28      [javac] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
23:09:28      [javac] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
23:09:28      [javac] Module=/usr/lib/system/libsystem_c.dylib
23:09:28      [javac] Module_base_address=00007FFF809D5000 Symbol=__findenv
23:09:28      [javac] Symbol_address=00007FFF80A34780
23:09:28      [javac] 
23:09:28      [javac] Method_being_compiled=openj9/internal/tools/attach/target/FileLock.unlockFileImpl(J)I
23:09:28      [javac] Target=2_90_20200218_292 (Mac OS X 10.11.6)
23:09:28      [javac] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
23:09:28      [javac] ----------- Stack Backtrace -----------
23:09:28      [javac] ---------------------------------------
pshipton picture pshipton on 19 Feb 2020

From an internal test Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/328

Testing: Run !shrc cpstats
Test start time: 2020/03/05 22:07:44 Eastern Standard Time
Running command: /Users/jenkins/workspace/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly/openjdkbinary/j2sdk-image/bin/jdmpview -core j9core1.dmp
Time spent starting: 2 milliseconds
Time spent executing: 6907 milliseconds
Test result: FAILED
Output from test:
 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x0005ffff
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
 [ERR] Handler1=000000000F2A2E20 Handler2=000000000E1A0A80
 [ERR] RDI=000000001078F7EF RSI=00000000B0153334 RAX=0000000000000000 RBX=000000001078F7EF
 [ERR] RCX=0000000000000000 RDX=00007FF9ABC1CB40 R8=00000000FFFFFFF3 R9=000000000000000D
 [ERR] R10=00007FF9ABC1CB40 R11=E000000000000000 R12=000000000E1419B8 R13=00000000B01539E8
 [ERR] R14=00000000FFFFFFF3 R15=0000000000000000
 [ERR] RIP=00007FFF9127BAA3 GS=0000 FS=0000 RSP=00000000B0153308
 [ERR] RFlags=0000000000010202 CS=002B RBP=00000000B0153320 ERR=4756A00000000000
 [ERR] TRAPNO=000000000000000D CPU=A000000000000000 FAULTVADDR=000000004756A000
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 000000057fffffff (f: 2147483648.000000, d: 1.167098e-313)
 [ERR] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM4 0000009600000320 (f: 800.000000, d: 3.182994e-312)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF9121D000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF9127BA49
 [ERR] 
 [ERR] Method_being_compiled=sun/nio/fs/UnixNativeDispatcher.getcwd()[B
 [ERR] Target=2_90_20200305_334 (Mac OS X 10.10.5)
 [ERR] CPU=amd64 (8 logical CPUs) (0x400000000 RAM)

To rebuild the failed tests in =https://ci.adoptopenjdk.net/job/Grinder, use the following links:
22:59:29 https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=8&JDK_IMPL=openj9&BUILD_LIST=functional&PLATFORM=x86-64_mac&TARGET=cmdLineTester_shrcdbgddrext_1

JasonFengJ9 picture JasonFengJ9 on 6 Mar 2020

Saw similar crash on internal build, java/lang/ProcessBuilder/Basic.java#id0.Basic_id0

>'255'< not equal to '0'
java.lang.Throwable
    at java.base/java.lang.Thread.dumpStack(Thread.java:511)
    at Basic.fail(Basic.java:2730)
    at Basic.fail(Basic.java:2731)
    at Basic.equal(Basic.java:2737)
    at Basic.testIORedirection(Basic.java:1039)
    at Basic.realMain(Basic.java:1269)
    at Basic.main(Basic.java:2740)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    at java.base/java.lang.Thread.run(Thread.java:834)
>'standard errorUnhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
Handler1=000000000489C380 Handler2=0000000003FB75A0
RDI=0000000005DD15BF RSI=00000000B0258374 RAX=0000000000000000 RBX=0000000005DD15BF
RCX=000000000000004A RDX=00007FA2B34670A0 R8=00000000FFFFFFF3 R9=000000000000000D
R10=00007FA2B34670C8 R11=6669746F4E686361 R12=000000002CF0A5F0 R13=000000001CE9F600
R14=00000000FFFFFFF3 R15=0000000000000054
RIP=00007FFF909ACAA3 GS=0000 FS=0000 RSP=00000000B0258348
RFlags=0000000000010202 CS=002B RBP=00000000B0258360 ERR=C1FEE00000000000
TRAPNO=000000000000000D CPU=E000000000000000 FAULTVADDR=00000001C1FEE000
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000f80100000014 (f: 20.000000, d: 1.347234e-309)
XMM2 000007d000000fa0 (f: 4000.000000, d: 4.243992e-311)
XMM3 2f6f692f6176616a (f: 1635148160.000000, d: 3.311397e-80)
XMM4 6544656c69462f6f (f: 1766207360.000000, d: 6.612054e+179)
XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/usr/lib/system/libsystem_c.dylib
Module_base_address=00007FFF9094E000 Symbol=__findenv
Symbol_address=00007FFF909ACA49

Method_being_compiled=java/io/FileDescriptor.close0()V
Target=2_90_20200320_371 (Mac OS X 10.10.5)
CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
----------- Stack Backtrace -----------
liqunl picture liqunl on 23 Mar 2020

assigned to @rpshukla

cathyzhyi picture cathyzhyi on 23 Mar 2020

https://ci.eclipse.org/openj9/job/Test_openjdk14_j9_sanity.functional_x86-64_mac_Release/8
J9vmTest_5
j9vm.test.monitor.CancelDeadThreadTest
vmState=0x0005ffff Symbol=__findenv

pshipton picture pshipton on 28 Mar 2020

Note the problem occurred a couple of time in grinders in a specific test, as shown in #9108.

pshipton picture pshipton on 3 Apr 2020

Note the problem occurred a couple of time in grinders in a specific test, as shown in #9108.

I'll try reproducing the failure locally for java/util/zip/EntryCount64k.java.EntryCount64k. If that works, then I'll try running it in valgrind or lldb to see if that yields any insights.

rpshukla picture rpshukla on 3 Apr 2020

Note that for a jdk_custom grinder, the name of the test is test/jdk/java/util/zip/EntryCount64k.java

pshipton picture pshipton on 3 Apr 2020
👍1

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.functional_x86-64_mac_Nightly/326
cmdLineTester_lockWordAlignment_Object_d_0

pshipton picture pshipton on 5 Apr 2020

Update on my investigation:

I haven't yet been able to reproduce this locally on my macbook. However, I did use dapptrace to log system libc calls on a successful run of test/jdk/java/util/zip/EntryCount64k.java. Note that I did not see any calls to setenv or putenv (something which could cause a seg fault in a multi-threaded setting). Using dapptrace, I noticed the symbol on my system was called __findenv_locked instead of __findenv.

Looking into the sources for libc on MacOS, I noticed that between libc version 1082.50.1 and 1158.1.2, the function __findenv was renamed to __findenv_locked. Looking at the libc code, the newer libc now seems to acquire a lock before calling __findenv_locked. Note that the libc on my laptop is version 1272.250.1 so it has the locked version.

I was also able to reproduce the failure in this grinder where I set the node to a machine with MacOS 10.10. 1 of 10 runs failed: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2830/

I am running a 100x grinder on MacOS 10.14 to see if the failure will show up on a newer system: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2831/

If the MacOS 10.14 grinder doesn't have any failures, then I suspect this seg fault is related to the libc version on the system. I'm not sure exactly how libc versions map to MacOS versions but the problem has definitely been seen on MacOS 10.10.

Update: the 100x MacOS 10.14 grinder had no failures.

rpshukla picture rpshukla on 16 Apr 2020
👍1

100x grinder for test/jdk/java/util/zip/EntryCount64k.java on MacOS 10.10 failed 6/100 runs
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2829/

rpshukla picture rpshukla on 16 Apr 2020

It looks like macOS 10.12 is the first version to have the __findenv_locked version of libc (Libc-1158.1.2): https://opensource.apple.com/release/macos-1012.html

10.11.6 has Libc-1082.60.1 which doesn't have source code available but some of the failures above occurred on a 10.11.6 machine so I'm guessing it has the non-locked version of __findenv: https://opensource.apple.com/release/os-x-10116.html

10.11.5 has Libc-1082.50.1 which definitely has the non-locked version of __findenv: https://opensource.apple.com/release/os-x-10115.html

I will look into trying to reproduce the failure on a machine with 10.11.6 or earlier.

rpshukla picture rpshukla on 16 Apr 2020

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.openjdk_x86-64_mac_Nightly/12
osx1011-x86-1
java/util/Locale/bcp47u/SystemPropertyTests.java.SystemPropertyTests

pshipton picture pshipton on 17 Apr 2020

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.openjdk_x86-64_mac_Nightly/13
osx1011-x86-2
tools/pack200/AttributeTests.java

pshipton picture pshipton on 20 Apr 2020

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.openjdk_x86-64_mac_Nightly/12
osx1011-x86-2
java/lang/ClassLoader/Assert.java

pshipton picture pshipton on 20 Apr 2020

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.openjdk_x86-64_mac_Nightly/17
osx1011-x86-1
java/util/zip/EntryCount64k.java

pshipton picture pshipton on 24 Apr 2020

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.openjdk_x86-64_mac_Nightly/18
osx1011-x86-2
java/lang/ProcessBuilder/Basic.java

pshipton picture pshipton on 26 Apr 2020

I鈥檝e been trying to find documentation from apple on the change from the unlocked to the locked versions of libc. So far I haven鈥檛 been able to find anything relevant. Looking at the source code, I think this lock was added to protect the static memory accessed by a call to a function called _NSGetEnviron.

Again, here is the source for the locked version of getenv : https://opensource.apple.com/source/Libc/Libc-1158.1.2/stdlib/FreeBSD/getenv.c.auto.html

And here is the source for _NSGetEnviron:
https://opensource.apple.com/source/Libc/Libc-1158.1.2/sys/crt_externs.c.auto.html

Note this bit of code inside getenv where the lock is acquired:

    __environ_lock();
    char *result = __findenv_locked(name, &offset, *_NSGetEnviron());
    __environ_unlock();

Furthermore, looking at the posix standard for getenv, this function does not need to be thread-safe: https://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html. The lock seems to have been added by apple as a convenience.

@andrewcraik fyi

rpshukla picture rpshukla on 30 Apr 2020

so it seems to me like they are serializing the access to make it thread safe (or more thread safe) and so the older versions are suffering from the lack of thread safety. I wonder what other APIs (apart from putenv/setenv etc) could modify NSGetEnviron - that might tell us what APIs we need to check for in a trace of system calls to see where the thread race might be.

I guess it is possible the get itself is causing some kind of race (if so only some kind of sycnhronizatoin like the above) is guaranteed to fix it.

andrewcraik picture andrewcraik on 30 Apr 2020
👍1

Looking through the libc sources, the only uses of _NSGetEnviron I see outside of getenv/putenv/setenv are to pass the environment as an argument to posix_spawn and exec. I don't see any function other than putenv/setenv that modifies the environment from _NSGetEnviron. Either something else is modifying the environment and I haven't found it or getenv itself is causing a race.

rpshukla picture rpshukla on 4 May 2020

@DanHeidinga or others familiar with the port library - we seem to be getting to the limits of what we can do. The great analysis from @rpshukla shows that the newer versions of OSX seem to have made getenv thread safe with a lock while the older versions are not. Now the getenv is not, in general, threadsafe in POSIX, but we have not had this issue on other platforms. A system level trace of the system calls from OpenJ9 did not show any of the usual suspects for corrupting the environment (setenv/putenv etc). Given this should we just put a lock (like the one in the newer libc's) on the getenv calls on the older OSX platforms and leave it at that to fix the functional issue? We aren't really sure what else we can do on the investigation.

andrewcraik picture andrewcraik on 19 May 2020

In looking further at _NSGetEnviron, I found this issue https://gitlab.haskell.org/ghc/ghc/issues/2458 which states:

Shared libraries and bundles on MacOS X Leopard don't have direct access to environ (i.e. extern char **environ), which is only available to the loader ld when a complete program is being linked. If direct access to environ is needed, the _NSGetEnviron() routine, defined in can be used to retrieve the address of environ at runtime (man environ(7) see last paragraph in section PROGRAMMING).

There are a couple of places where environ is used in OpenJ9 (rasdump.c) & OMR (port/unix/omrsysinfo.c) which we can modify to use _NSGetEnviron instead (plus some zOS-specific uses).

Given this should we just put a lock (like the one in the newer libc's) on the getenv calls on the older OSX platforms and leave it at that to fix the functional issue?

This is worth trying. We can add code to lock / unlock around getenv calls in the JIT and see if that's sufficient to avoid the crash. Running that way for a month or so should give us a fair bit of confidence in whether or not the crash still occurs

DanHeidinga picture DanHeidinga on 20 May 2020

I'm looking into locking getenv calls and swapping environ with _NSGetEnviron.

sharon-wang picture sharon-wang on 28 May 2020

@pshipton Has this crash been seen recently (since April 26th)?

I've run 400+ grinder iterations, but I haven't had luck reproducing.

  • macOS versions tested: 10.10, 10.11
  • jdk_custom tests run:

    • test/jdk/java/lang/ClassLoader/Assert.java

    • test/jdk/java/util/zip/EntryCount64k.java

    • test/jdk/java/lang/ProcessBuilder/Basic.java

sharon-wang picture sharon-wang on 5 Jun 2020

Has this crash been seen recently (since April 26th)?

No. There is a second Issue for this problem (for no good reason) #5153 but the last reported occurrence in there is April 17.

pshipton picture pshipton on 5 Jun 2020
👍1

Still not able to reproduce in grinder. I've started running grinders with some additional options, i.e. -Xjit:count=0,optlevel=scorching -XcompilationThreads7 in hopes of increasing the chance of seeing the error. I'll be varying the optlevel as well.

Alongside testing, I'm adding locking around the j9sysinfo_get_env() calls in the JIT. Hopefully I can successfully reproduce the crash so that I can test these changes.

Also wanted to note that https://github.com/eclipse/openj9/issues/5061 looks like the same problem and has some good discussion there too.

sharon-wang picture sharon-wang on 15 Jun 2020

Ran several more sets of grinders, but no crashes occurred.

Tested with the following

  • OSX 10.10, 10.11
  • JVM_OPTIONS -Xjit:count=0,optlevel=scorching -XcompilationThreads7
  • Tests

    • testvmcheck_6

    • J9vmTest

    • test/jdk/java/lang/ClassLoader/Assert.java

    • test/jdk/java/util/zip/EntryCount64k.java

    • test/jdk/java/lang/ProcessBuilder/Basic.java

Seems like this issue is not reproducible at the moment...if this crash shows up again, I'll circle back to this.

sharon-wang picture sharon-wang on 19 Jun 2020

I've opened https://github.com/eclipse/openj9/pull/9994 and https://github.com/eclipse/omr/pull/5338 to replace environ with _NSGetEnviron() for OSX

sharon-wang picture sharon-wang on 24 Jun 2020

Another _findEnv crash with a different vmState.

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_mac_OMR_testList_0/42
cmdLineTester_callsitedbgddrext_openj9_0
Run !printallcallsites

 [ERR] Unhandled exception
 [ERR] Type=Segmentation error vmState=0x00050bb0
 [ERR] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
 [ERR] Handler1=0000000007AF15D0 Handler2=0000000007CDD680
 [ERR] RDI=00000000090F4F44 RSI=0000700000254F14 RAX=0000000000000000 RBX=00000000090F4F44
 [ERR] RCX=0000000000000000 RDX=00007FF36B413EB0 R8=00000000FFFFFFD5 R9=000000000000002B
 [ERR] R10=00007FF36B413EB0 R11=C000000000000000 R12=0000000007A2CAE8 R13=000000003D913F20
 [ERR] R14=0000000000000000 R15=0000000000000000
 [ERR] RIP=00007FFF9C7777D4 GS=0000 FS=0000 RSP=0000700000254EE0
 [ERR] RFlags=0000000000010206 CS=002B RBP=0000700000254F00 ERR=090F4F4400000000
 [ERR] TRAPNO=000000000000000D CPU=4F44000000000000 FAULTVADDR=00000000090F4F44
 [ERR] XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM2 0000000028e22d90 (f: 685911424.000000, d: 3.388853e-315)
 [ERR] XMM3 54676e6979706f43 (f: 2037411712.000000, d: 4.003916e+98)
 [ERR] XMM4 646e494f54525467 (f: 1414681728.000000, d: 5.992591e+175)
 [ERR] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [ERR] Module=/usr/lib/system/libsystem_c.dylib
 [ERR] Module_base_address=00007FFF9C718000 Symbol=__findenv
 [ERR] Symbol_address=00007FFF9C777780
 [ERR] 
 [ERR] Method_being_compiled=java/lang/String.indexOf(II)I
 [ERR] Target=2_90_20200706_643 (Mac OS X 10.11.6)
 [ERR] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
 [ERR] ----------- Stack Backtrace -----------
 [ERR] ---------------------------------------
pshipton picture pshipton on 7 Jul 2020

@AdamBrousseau @jdekonin Can you confirm the version of OSX installed on https://ci.eclipse.org/openj9/computer/osx1011-x86-1/ ? We've been trying to hunt down this __findenv crash and the current theory is that's related to the OS level but we need confirmation about which version of osx the test was running on.

DanHeidinga picture DanHeidinga on 7 Jul 2020 
osx1011vm1:~ jenkins$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G22010

Machine -2 is the same.

Edit:
Internal, osxrt5 10.11.6, osxrt[8,13,14] 10.10.5.

AdamBrousseau picture AdamBrousseau on 7 Jul 2020
👍1

Ran some more grinders on osxrt5; haven't reproduced yet. Will vary JIT-related JVM options to try to reproduce.

  • 70x cmdLineTester_callsitedbgddrext_openj9_0
  • 40x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=scorching
  • 20x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=veryHot
  • 20x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=hot
  • 20x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=warm
  • 20x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=cold
  • 20x cmdLineTester_callsitedbgddrext_openj9_0 -Xjit:count=0,-Xjit:optlevel=noOpt

I don't have access to run grinders on the ci.eclipse machines, so here's some attempts to hack together the grinder URLs if a committer would like to launch them :)

osx1011-x86-1 grinder build link, osx1011-x86-2 grinder build link

In case those links don't work, these are the parameters I'm using:

PLATFORM x86-64_mac
BUILD_LIST functional
TARGET cmdLineTester_callsitedbgddrext_openj9_0
LABEL osx1011-x86-1 OR osx1011-x86-2
SDK_RESOURCE nightly
ITERATIONS 10

Could also run some grinders on the adopt machines, but it seems there isn't an OSX machine with version 10.11 https://ci.adoptopenjdk.net/label/sw.os.osx/.

sharon-wang picture sharon-wang on 7 Jul 2020

10x osx1011-x86-1 https://ci.eclipse.org/openj9/job/Grinder/934
10x osx1011-x86-2 https://ci.eclipse.org/openj9/job/Grinder/935

pshipton picture pshipton on 7 Jul 2020

20x osx1011-x86-1 https://ci.eclipse.org/openj9/job/Grinder/936
20x osx1011-x86-2 https://ci.eclipse.org/openj9/job/Grinder/937
20x osx1011-x86-1 https://ci.eclipse.org/openj9/job/Grinder/938
20x osx1011-x86-2 https://ci.eclipse.org/openj9/job/Grinder/939
20x osx1011-x86-1 https://ci.eclipse.org/openj9/job/Grinder/940
20x osx1011-x86-2 https://ci.eclipse.org/openj9/job/Grinder/941

pshipton picture pshipton on 7 Jul 2020

fyi https://github.com/eclipse/openj9/issues/11430

pshipton picture pshipton on 10 Dec 2020

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Release_testList_1/19
testDDRExtJunit_MonitorsAndDeadlock3_1
osx1011-x86-2

TCK.run.tests.ddrext
...
[java] Unhandled exception
     [java] Type=Segmentation error vmState=0x000503ff
     [java] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
     [java] Handler1=000000000D699420 Handler2=000000000CCCB040
     [java] RDI=000000000EB7E9E6 RSI=0000700000253E34 RAX=0000000000000000 RBX=000000000EB7E9E6
     [java] RCX=0000000000000000 RDX=00007FEC30C00470 R8=00000000FFFFFFE8 R9=0000000000000018
     [java] R10=00007FEC30C00470 R11=2000000000000000 R12=000000002F90E48E R13=000000001E694A80
     [java] R14=0000000000000000 R15=0000000000000000
     [java] RIP=00007FFF843987D4 GS=0000 FS=0000 RSP=0000700000253E00
     [java] RFlags=0000000000010206 CS=002B RBP=0000700000253E20 ERR=0EB7E9E600000000
     [java] TRAPNO=000000000000000D CPU=E9E6000000000000 FAULTVADDR=000000000EB7E9E6
     [java] XMM0 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
     [java] XMM1 000000003dcccccd (f: 1036831936.000000, d: 5.122630e-315)
     [java] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM3 6974752f6176616a (f: 1635148160.000000, d: 9.787124e+199)
     [java] XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] Module=/usr/lib/system/libsystem_c.dylib
     [java] Module_base_address=00007FFF84339000 Symbol=__findenv
     [java] Symbol_address=00007FFF84398780
     [java] 
     [java] Method_being_compiled=sun/misc/URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
     [java] Target=2_90_20210102_54 (Mac OS X 10.11.6)
     [java] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
     [java] ----------- Stack Backtrace -----------
     [java] ---------------------------------------
pshipton picture pshipton on 4 Jan 2021

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_x86-64_mac_Nightly_testList_1/241
testDDRExt_Callsites_0
osx1011-x86-2

     [java] Unhandled exception
     [java] Type=Segmentation error vmState=0x000503ff
     [java] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000
     [java] Handler1=0000000007A33820 Handler2=0000000007123AA0
     [java] RDI=00000000090723C7 RSI=0000700000255294 RAX=0000000000000000 RBX=00000000090723C7
     [java] RCX=000000000000004A RDX=00007FC312F00480 R8=00000000FFFFFFE8 R9=0000000000000018
     [java] R10=00007FC312F00788 R11=0031000000000000 R12=00000000FFFFFFE8 R13=0000000028FB7720
     [java] R14=000000000000004A R15=0000000000000054
     [java] RIP=00007FFF843987D4 GS=0000 FS=0000 RSP=0000700000255260
     [java] RFlags=0000000000010206 CS=002B RBP=0000700000255280 ERR=080BC60000000000
     [java] TRAPNO=000000000000000D CPU=C600000000000000 FAULTVADDR=00000000080BC600
     [java] XMM0 0000000000001000 (f: 4096.000000, d: 2.023693e-320)
     [java] XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM3 6974752f6176616a (f: 1635148160.000000, d: 9.787124e+199)
     [java] XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
     [java] Module=/usr/lib/system/libsystem_c.dylib
     [java] Module_base_address=00007FFF84339000 Symbol=__findenv
     [java] Symbol_address=00007FFF84398780
     [java] 
     [java] Method_being_compiled=sun/misc/URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
     [java] Target=2_90_20210201_641 (Mac OS X 10.11.6)
     [java] CPU=amd64 (4 logical CPUs) (0x200000000 RAM)
     [java] ----------- Stack Backtrace -----------
     [java] ---------------------------------------
pshipton picture pshipton on 2 Feb 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

xliang6 picture xliang6  路  3Comments
xliang6 picture xliang6  路  6Comments
JasonFengJ9 picture JasonFengJ9  路  5Comments
JamesKingdon picture JamesKingdon  路  5Comments
pshipton picture pshipton  路  3Comments