Openj9: Initial functional test results against openj9 osx build (jdk9)

[smlambert]

Will start recording initial findings in this issue.

Some tests hang, so the set of tests have to be run is small groups, not sanity.functional or extended.functional as those large targets both contain tests that hang.

Testing with: java -version
openjdk version "9.0.4-internal"
OpenJDK Runtime Environment (build 9.0.4-internal+0-adhoc.nazim.nasm-osx-pr-build)
Eclipse OpenJ9 VM (build nasm-initial-wip-b067ee64, JRE 9 Darwin amd64-64-Bit 20181001_000000 (JIT enabled, AOT enabled)
OpenJ9 - b067ee64
OMR - fe8efc6c
JCL - 361429b612 based on jdk-9.0.4+12)

---

Test targets that fail:
AttachAPISanity_0 - attachapisanity_report.html.txt #3206
jsr292Test_0, jsr292Test_1 - jsr292Test_1_report.html.txt @fengxue-IS Is looking at these
JLM_Tests_interface - JLM_Tests-report.html.txt @JasonFengJ9 is looking at these
SoftmxRASTest1, SoftmxRASTest2 - SoftmxRASTest_report.html.txt
J9vmTest - all fail, TestTargetResult_J9vmTest.tap.txt @pdbain-ibm looked at these
(as expected with Darwin is not a supported OS platform error)

in cmdLineTests dir:
(Sanity: TOTAL: 81 EXECUTED: 81 PASSED: 2 FAILED: 79 SKIPPED: 0)
(Extended: TOTAL: 83 EXECUTED: 83 PASSED: 0 FAILED: 83 SKIPPED: 0)
Most cmdLineTests fail with " Darwin is not a supported OS platform"
Some variants of cmdLineTester_GCRegressionTests - fail with JVMJ9VM007E Command-line option unrecognised: -Xgcpolicy:metronome - @pdbain-ibm can you look at this?
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

---

Test targets that hang:
jsr292Test_JitCount0_0
IllegalAccessProtectedMethodTest_0
jit_jitt_0 (hangs at cfgTest subtest)
VarHandleInvokerTest_1

---

To reproduce (install test prereqs on machine if not present):

1. Unpack jdk9 openj9 osx SDK to /someLocation
2. git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
3. cd openjdk-tests
4. ./get.sh -t /testLocation/openjdk-tests -p x64_mac -i openj9
5. cd TestConfig
6. export environment variables ( export JAVA_BIN=/someLocation/bin export SPEC=macos_x86-64 export JDK_VERSION=9 export JDK_IMPL=openj9 export BUILD_LIST=functional ) Reminder: JDK_VERSION=11, if you are testing a JDK11 build.
7. make -f run_configure.mk
8. make compile (DDR_Test requires ddrtools to compile, if your build does not have that, quick workaround is to delete openjdk-tests/functional/DDR_Test directory, all other test dirs compile)
9. make _AttachAPISanity_0 (or any other of the failing targets listed above, prepended with _)

Note: to run all sanity tests inside a directory of tests, for step 9, can use:
make -C ../functional/UnsafeTest -f autoGen.mk _sanity
or _extended to run all extended tests in the dir

[0xdaryl]

Thanks @smlambert. In my own private testing, any JIT test run without -Xrs appears to hang on NPEs or arithmetic exceptions (these could be programmatic exceptions). We'll have to triage the JIT mode hangs with that option to be sure.

[smlambert]

This will take a while, I will keep updating this single issue, and leave it to others to divide and conquer from this initial list (creating other issues for items being worked).

[DanHeidinga]

Test targets that fail:
AttachAPISanity_0 - attachapisanity_report.html.txt

@pdbain-ibm Can you take a look at these tests?

[DanHeidinga]

Test targets that fail:
jsr292Test_0, jsr292Test_1 - jsr292Test_1_report.html.txt

@fengxue-IS Can you take a look at these tests?

[pdbain-ibm]

In re: https://github.com/eclipse/openj9/issues/3154#issuecomment-427220987
the problem is with java.lang.ProcessBuilder:

Caused by: java.lang.Error: Darwin is not a supported OS platform.
    at java.base/java.lang.ProcessImpl$Platform.get(ProcessImpl.java:165)
    at java.base/java.lang.ProcessImpl.<clinit>(ProcessImpl.java:169)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at java.base/java.lang.Runtime.exec(Runtime.java:635)
    at java.base/java.lang.Runtime.exec(Runtime.java:494)
    at org.openj9.test.attachAPI.TargetManager.launchTarget(TargetManager.java:248)

Note that the attach API tests by necessity launch child processes. This issue will probably also kill the VmArgument tests.

[smlambert]

Put passing tests report here (to declutter description above), but still track which directories of tests have been run:

Test targets that pass:
JCL_Test_0 (GeneralTest suite)

• JCL_TEST_Test-Annotation-Package
• JCL_TEST_Test-Annotation-Identity
• JCL_TEST_Test-Annotation-ClassLoader
• JCL_TEST_Test-TypeAnnotation
• JCL_TEST_Test-Annotation

SharedCPEntryInvokerTests_1
regressionFastresolve_mode150

JIT_Test dir:

• finalizerTest
• stringConcatOptTest
• jit_jitt_0 (passes arrayTest subtest)
• jit_jitt_0 (passes assemblerTest subtest)
• jit_jitt_0 (passes castingTest subtest)

JLM_Tests dir:

• JLM_Tests_class
• JLM_Tests_IBMinternal

Java9AndUp dir:

• MethodVisibilityTests
• UnreflectTests
• StackWalkerTest
• CallerSensitiveGetCallerClassTest
• VarHandleInvokerTest_0

JavaAgentTest dir:
TestFlushReflectionCache
TestRefreshGCSpecialClassesCache_NOBCI

Jsr335 & Jsr335_interfaceStaticMethod dir:
jsr335tests
jsr335tests_defendersupersends
jsr335_interfaceStaticMethod

OpenJ9_Jsr_292_API dir:
openj9_jsr292Test
openj9_jsr292Test_JitCount0

SharedCPEntryInvokerTests dir:
SharedCPEntryInvokerTests_0

UnsafeTest dir:
UnsafeTests

VM_Test dir:
ThreadRegressionTests

cmdLineTests dir:
cmdLineTester_jrvTest
decompileAtMethodResolve

[smlambert]

I should note, currently the DDR_Test directory does not compile, as it seems to require j9ddr tools that are not available from the developer build I am using (the quick workaround is to delete it):

[javac] Compiling 51 source files to /Users/nazim/Desktop/WIP/openjdk-tests/functional/DDR_Test/bin
[javac] /Users/nazim/Desktop/WIP/openjdk-tests/functional/DDR_Test/src/j9vm/test/ddrext/AutoRun.java:60: error: package com.ibm.j9ddr.tools.ddrinteractive does not exist
[javac] import com.ibm.j9ddr.tools.ddrinteractive.Context;
[javac]                                          ^
[javac] /Users/nazim/Desktop/WIP/openjdk-tests/functional/DDR_Test/src/j9vm/test/ddrext/AutoRun.java:310: error: cannot find symbol
[javac]     public static void runTest(Context ctx, PrintStream out, String testCaseList) {
[javac]                                ^
[javac]   symbol:   class Context
[javac]   location: class AutoRun
[javac] /Users/nazim/Desktop/WIP/openjdk-tests/functional/DDR_Test/src/j9vm/test/ddrext/DDRExtTesterBase.java:39: error: package com.ibm.j9ddr.tools.ddrinteractive does not exist

[nbhuiyan]

I have been trying to reproduce the hanging targets, and investigating whether that happens with using -Xrs.

jit_jitt_0

Using -Xrs option, I have been able to get past the hang in target jit_jitt_0.

As the script continued past that previously hanging point, I encountered another hang at jit_jitt_3 at cfgTest subset. Again, using -Xrs allowed me to get past that point all the way to completion of the sanity tests:

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
    finalizerTest_0
    stringConcatOptTest_0
    jit_jitt_0
    jit_jitt_1
    jit_jitt_2
    jit_jitt_3
    jit_jar_0
    jit_tr_0
    StringPeepholeTest_0
    jit_vich_0

TOTAL: 14   EXECUTED: 10   PASSED: 10   FAILED: 0   SKIPPED: 4
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

_sanity done

VarHandleInvokerTest_1

No longer hangs after using -Xrs and passes.

jsr292Test_JitCount0_0

Using -Xrs, this test did not hang, but resulted in a failure due to illegal access exceptions.

IllegalAccessProtectedMethodTest_0

No longer hangs after using -Xrs and passes.

[smlambert]

Thanks @nbhuiyan!

[0xdaryl]

@smlambert reports above that "openj9_jsr292Test_JitCount0" passes, yet @nbhuiyan observes a failure with "jsr292Test_JitCount0". Are these the same test?

[nbhuiyan]

Further info regarding jsr292Test_JitCount0_0: fails despite trying with optLevel=noOpt and -Xint mode.

[fengxue-IS]

Re comment

Test targets that fail:
jsr292Test_0, jsr292Test_1 - jsr292Test_1_report.html.txt

likely related to #3047 , still working on a fix.

@0xdaryl I believe these two tests are different, "openj9_jsr292Test_JitCount0" is under "OpenJ9_Jsr_292_API" which is sanity, and "jsr292Test_JitCount0" is under extended "Jsr292" tests

[fengxue-IS]

@nbhuiyan for jsr292Test_JitCount0_0, I believe that is the same issue as jsr292Test_0, see my comment above ^^^

[DanHeidinga]

likely related to #3047 , still working on a fix.

That's somewhat surprising if the tests pass on other platforms. Why is mac more sensitive to the checkClassAccess call?

[JasonFengJ9]

Looking at failure: JLM_Tests_interface - JLM_Tests-report.html.txt.

[pdbain-ibm]

Looking at Some variants of cmdLineTester_GCRegressionTests - fail with JVMJ9VM007E Command-line option unrecognised: -Xgcpolicy:metronome

[JasonFengJ9]

There are three AssertionError within JLM_Tests_interface - JLM_Tests-report.html.txt.

Two of them, testGetCurrentThreadUserTime and testGetThreadUserTime are due to OSX is not supported within thrprof.c:omrthread_get_user_time.

Here is how the testcase failed:

FAILED: testGetCurrentThreadUserTime
java.lang.AssertionError: 
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:33)
    at org.openj9.test.java.lang.management.TestThreadMXBean.testGetCurrentThreadUserTime(TestThreadMXBean.java:282)     --> AssertJUnit.assertTrue(tb.getCurrentThreadUserTime() > -1);

com.ibm.java.lang.management.internal.ThreadMXBeanImpl.getCurrentThreadUserTime() --> native getThreadUserTimeImpl(id) -->  mgmtthread.c: Java_com_ibm_java_lang_management_internal_ThreadMXBeanImpl_getThreadUserTimeImpl() --> getCurrentThreadUserTime(id) --> omrthread_get_self_user_time()  --> thrprof.c:  omrthread_get_user_time(self)  --> omrthread_get_user_time(id)

Similarly,

FAILED: testGetThreadUserTime
java.lang.AssertionError: 
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24)
    at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:33)
    at org.openj9.test.java.lang.management.TestThreadMXBean.testGetThreadUserTime(TestThreadMXBean.java:481)  --> AssertJUnit.assertTrue(tb.getThreadUserTime(Thread.currentThread().getId()) > -1);

com.ibm.java.lang.management.internal.ThreadMXBeanImpl.getThreadUserTime() --> native getThreadUserTimeImpl(id)

Still looking at another AssertionError

FAILED: testGetMBeanInfo
java.lang.AssertionError: expected:<14> but was:<16>
    at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
    at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
    at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
    at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
    at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
    at org.openj9.test.java.lang.management.TestThreadMXBean.testGetMBeanInfo(TestThreadMXBean.java:1290)   ---> AssertJUnit.assertEquals(14, operations.length);

Update: testGetMBeanInfo AssertionError is a known failure https://github.com/eclipse/openj9/issues/1626

[fengxue-IS]

likely related to #3047 , still working on a fix.

That's somewhat surprising if the tests pass on other platforms. Why is mac more sensitive to the checkClassAccess call?

Re tested with latest build/test material, the test failure is fixed, looks like a miss match between test/vm changes

[JasonFengJ9]

Looking at SoftmxRASTest1, SoftmxRASTest2 - SoftmxRASTest_report.html.txt

[JasonFengJ9]

j9vm.test.softmx.SoftmxRASTest1#testSoftmx_RAS_Test_1

java.lang.Error: Darwin is not a supported OS platform.
    at java.base/java.lang.ProcessImpl$Platform.get(ProcessImpl.java:165)
    at java.base/java.lang.ProcessImpl.<clinit>(ProcessImpl.java:169)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
    at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
    at java.base/java.lang.Runtime.exec(Runtime.java:635)
    at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58)

similar with https://github.com/eclipse/openj9/issues/3206. Will test with latest build which contains https://github.com/eclipse/omr/pull/3053.

[JasonFengJ9]

Re-run the test _SoftmxRASTest1 and _SoftmxRASTest2 with latest build:
Now some sub-test pass and others failed with following errors:

This JVM package only includes the '-Xcompressedrefs' configuration. Please run the VM without specifying the '-Xnocompressedrefs' option or by specifying the '-Xcompressedrefs' option.
To compile the other configuration, please run configure with '--with-noncompressedrefs.

These are known issues due to no support to macos_x86-64_cmpptrs yet.

[DanHeidinga]

These are known issues due to no support to macos_x86-64_cmpptrs yet.

The specs for compressed refs have been merged. Can you validate that this builds and runs with a compressedrefs build?

[pdbain-ibm]

Looking at Some variants of cmdLineTester_GCRegressionTests - fail with JVMJ9VM007E Command-line option unrecognised: -Xgcpolicy:metronome

I tested make JVM_OPTIONS=-Xcompressedrefs _cmdLineTester_GCRegressionTests_0, all passed. @smlambert could you be more specific about which variants of the test? A copy of the command line and/or console log would help. Thanks.

[JasonFengJ9]

The failure wasn't due to VM configuration. The test should skip incompatible tests, i.e., skip -Xcompressedrefs tests for noncompressedrefs JVM, or skip -Xnoncompressedrefs tests for compressedrefs JVM.

Looked at same SoftmxRASTest1 for linux_x86-64_cmprssptrs spec, within functional/JLM_Tests/autoGen.mk

SoftmxRASTest1_0_INVALID_PLATFORM_CHECK=$(filter linux_x86-64_cmprssptrs, $(SPEC))
ifeq ($(SoftmxRASTest1_0_INVALID_PLATFORM_CHECK),)
    @echo "test with Mode110"
else
@echo "Skipped due to jvm options ($(JVM_OPTIONS)) and/or platform requirements (^os.zos,bit.64) => $(TEST_SKIP_STATUS)" | tee -a $(Q)$(TESTOUTPUT)$(D)TestTargetResult$(Q);
endif
TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
    SoftmxRASTest1_2

TOTAL: 4   EXECUTED: 1   PASSED: 1   FAILED: 0   SKIPPED: 3
ALL TESTS PASSED

But for osx_x86-64_cmprssptrs there was no SoftmxRASTest1_0_INVALID_PLATFORM_CHECK, and all sub-test run regardless of JVM configuration.

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
    SoftmxRASTest1_2 .  --> pass with osx_x86-64_cmprssptrs

FAILED test targets:
    SoftmxRASTest1_0 .  --> these three tests with -Xnocompressedrefs should be skipped
    SoftmxRASTest1_1
    SoftmxRASTest1_3

TOTAL: 4   EXECUTED: 4   PASSED: 1   FAILED: 3   SKIPPED: 0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[JasonFengJ9]

Thanks @smlambert https://github.com/eclipse/openj9/issues/3307

[smlambert]

No official builds yet, but uploaded dev build to Adopt to more easily run different groups of tests:

JDK11 OpenJ9 OSX Developer build (from @keithc-ca, thanks!)

"/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -version
openjdk version "11-internal" 2018-09-25
OpenJDK Runtime Environment (build 11-internal+0-adhoc.keithc.jdk11)
Eclipse OpenJ9 VM (build master-1e9f12d6a1, JRE 11 Mac OS X amd64-64-Bit Compressed References 20181016_000000 (JIT enabled, AOT enabled)
OpenJ9 - 1e9f12d6a1
OMR - 77f4df76
JCL - 4f9e2ba5c4 based on jdk-11+28)

with _sanity.openjdk test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/305/
with _sanity.system test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/306/ - openj9-systemtests fail to compile (issue https://github.com/eclipse/openj9-systemtest/issues/49)
with _sanity.functional test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/307/

[pdbain-ibm]

Working on the cmdlinetest jobs under https://github.com/eclipse/openj9/issues/3321.

[JasonFengJ9]

With latest https://github.com/eclipse/openj9/blob/master/test/TestConfig/resources/ottawa.csv, SoftmxRASTest1, SoftmxRASTest2 now pass.

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
    SoftmxRASTest1_2

TOTAL: 4   EXECUTED: 1   PASSED: 1   FAILED: 0   SKIPPED: 3
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PASSED test targets:
    SoftmxRASTest2_2

TOTAL: 4   EXECUTED: 1   PASSED: 1   FAILED: 0   SKIPPED: 3
ALL TESTS PASSED
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[pdbain-ibm]

Some variants of cmdLineTester_GCRegressionTests - fail with JVMJ9VM007E Command-line option unrecognised: -Xgcpolicy:metronome - @pdbain-ibm can you look at this?
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Could not reproduce the issue: see https://github.com/eclipse/openj9/issues/3321. Note that I ran with JVM_OPTIONS=-Xcompressedrefs. Please advise if there are different modes I should try.

[JasonFengJ9]

with _sanity.openjdk test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/305/

TAP Extended Test Results: 1 files | 7 tests, 3 ok, 4 not ok, 0 skipped, 0 ToDo, 0 Bail Out!
There wasn't enough info about those 4 not ok tests.

with _sanity.system test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/306/ - openj9-systemtests fail to compile (issue eclipse/openj9-systemtest#49)
with _sanity.functional test target: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/307/

_sanity.functional failures are due to compilation failure as well (native tests weren't built).

[pdbain-ibm]

Problems compiling system tests: https://github.com/eclipse/openj9-systemtest/issues/52.

[smlambert]

Details for _sanity.openjdk tests (30 out of 3,060 tests fail) are also available as junit results (not just TAP): https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/305/testReport/
and also more details available via .jtr files if you unzip in the archived results.

[JasonFengJ9]

Looking at java/lang/System/MacEncoding/MacJNUEncoding.java.MacJNUEncoding.

[pdbain-ibm]

Looked at java/lang/StackWalker/LocalsAndOperands.java#id0.LocalsAndOperands_id0 and
java/lang/StackWalker/LocalsAndOperands.java#id1.LocalsAndOperands_id1:

TEST: java/lang/StackWalker/LocalsAndOperands.java#id1
TEST JDK: /Users/pdbain/vm/openj9-openjdk-jdk11/build/macosx-x86_64-normal-server-release/images/jdk/bin/..

TEST RESULT: Error. Parse Exception: Syntax error in @requires expression: invalid name: vm.graal.enabled

This looks Hotspot-specific, right @pshipton ?

TEST: java/lang/StackWalker/LocalsAndOperands.java#id0
Caused by: java.lang.ClassNotFoundException: java.lang.LiveStackFrame

LiveStackFrame is not public API.

Can we exclude these tests for OpenJ9?

[pshipton]

More information is required. The first failure related to vm.graal.enabled should be able to run. The property should be set to false, not invalid. Similar problem to https://github.com/eclipse/openj9/issues/3048

Although LiveStackFrame is not public API, somehow the test runs and passed on Hotspot (I assume).

[pdbain-ibm]

LiveStackFrame is a package private interface in java.lang used in Hotspot's implementation of StackWalker. The test is relying on an unspecified implementation detail.

[pshipton]

I see, I thought OpenJ9 also had a LiveStackFrame class, but I see its just an empty file. I'm ok to exclude this one.

[pdbain-ibm]

Does this cover any unresolved issues or can we close it?

[AdamBrousseau]

Can this be closed?

[smlambert]

Yes, closing.