Openjdk-tests: Intermittent test failures list

Created on 16 Dec 2019  路  25Comments  路  Source: AdoptOpenJDK/openjdk-tests

So me and @M-Davies have been tracking any intermittent failing tests from https://ci.adoptopenjdk.net/ for a little while now, but this should be displayed publicly so that it can be seen and contributed by anyone else here.
Once a test fails more than a couple times, we can then investigate by looking which platforms/versions etc it failed on.

bug

All 25 comments

java/lang/ProcessBuilder/Basic.java#id0 jdk13-j9-ppc64le-linux 02-Dec-2019

Program '/home/jenkins/workspace/Test_openjdk13_j9_sanity.openjdk_ppc64le_linux/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 2400000ms, elapsed time including timeout handling was 2402121ms).

java/lang/invoke/MethodHandlesSpreadArgumentsTest.java
jdk11-j9-x64-windows 03-Dec-2019
jdk11-j9-x64-windows 16-Dec-2019
jdk11-j9-x64-windows 19-Dec-2019

RASIED: https://github.com/eclipse/openj9/issues/8208

junit -XX:+IgnoreUnrecognizedVMOptions -XX:-VerifyDependencies -esa test.java.lang.invoke.MethodHandlesSpreadArgumentsTest
Unhandled exception
Type=Segmentation error vmState=0x00000000
Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFD2D4EB95D ContextFlags=0010005f
...
TEST RESULT: Failed. Unexpected exit from test [exit code: -1]

java/net/httpclient/RestrictedHeadersTest.java jdk13-j9-x64-mac 03-Dec-2019

Error: Port Library failed to initialize: -1
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
TEST RESULT: Failed. Unexpected exit from test [exit code: 1]

java/util/zip/ZipFile/TestCleaner.java
jdk11-j9-x64-mac 04-Dec-2019
jdk11-j9-s390x-linux 04-Dec-2019
jdk11-j9-s390x-linux 12-Dec-2019
jdk11-j9-x64-macXL 19-Dec-2019

Agent error: java.lang.Exception: Agent 0 timed out with a timeout of 960 seconds; check console log for any additional details
Running jstack on process 45583
2019-12-04T18:09:21.047972
Error getting data from 45583: Exception connecting to 45583

java/io/RandomAccessFile/OpenSync.java jdk11-j9-s390x-linux 12-Dec-2019

Mode "d": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "d" must be one of "r", "rw", "rws", or "rwd"
Mode "s": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "s" must be one of "r", "rw", "rws", or "rwd"
Mode "rd": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "rd" must be one of "r", "rw", "rws", or "rwd"
Mode "rs": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "rs" must be one of "r", "rw", "rws", or "rwd"
Mode "rwx": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "rwx" must be one of "r", "rw", "rws", or "rwd"
Mode "foo": Thrown as expected: java.lang.IllegalArgumentException
  Illegal mode "foo" must be one of "r", "rw", "rws", or "rwd"
java.lang.NullPointerException
    at java.base/java.io.DataOutputStream.writeUTF(DataOutputStream.java:353)
    at java.base/java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1177)
    at OpenSync.main(OpenSync.java:60)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
    at java.base/java.lang.Thread.run(Thread.java:834)

java/net/httpclient/ShortResponseBody.java jdk11-j9-ppc64le-linuxXL 12-Dec-2019

Test Failure unclear. 1/379 failed

java/util/regex/RegExTest.java jdk13-j9-x64-linux 17-Dec-2019

07:12:03  Invalid compile flags         : Passed
07:12:03  java.lang.NullPointerException
07:12:03    at java.base/java.util.regex.Pattern.peek(Pattern.java:1892)
07:12:03    at java.base/java.util.regex.Pattern.sequence(Pattern.java:2118)
07:12:03    at java.base/java.util.regex.Pattern.expr(Pattern.java:2068)
07:12:03    at java.base/java.util.regex.Pattern.compile(Pattern.java:1782)
07:12:03    at java.base/java.util.regex.Pattern.<init>(Pattern.java:1429)
07:12:03    at java.base/java.util.regex.Pattern.compile(Pattern.java:1069)
07:12:03    at RegExTest.embeddedFlags(RegExTest.java:4763)
07:12:03    at RegExTest.main(RegExTest.java:183)
07:12:03    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
07:12:03    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
07:12:03    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
07:12:03    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
07:12:03    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
07:12:03    at java.base/java.lang.Thread.run(Thread.java:847)
07:12:26  [2019-12-17 07:12:24,044] Agent[0]: stderr: JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2019/12/17 07:12:24 - please wait.

openjdk11_hs_sanity.openjdk_aarch64_linux, VarHandles tests:

    java/lang/invoke/VarHandles/VarHandleTestAccessBoolean.java.VarHandleTestAccessBoolean
    java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessLong.java.VarHandleTestMethodHandleAccessLong
    java/lang/invoke/VarHandles/VarHandleTestMethodHandleAccessShort.java.VarHandleTestMethodHandleAccessShort

sun/net/www/AuthHeaderTest.java jdk8-j9-s390x-linux 18-Dec-2019

Server: listening on port: 35395
client opening connection to: http://localhost:35395/d1/foo.html
SERVER: new connection. chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56124]]
SERVER: connection readable. chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56124]]
Basic dXNlcjpwYXNzd29yZE5vdENoZWNrZWRBbnl3YXk=
SERVER: orderlyCloseChannel chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56124]]
SERVER: connection readable. chan[java.nio.channels.SocketChannel[connected oshut local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56124]]
Server: comsume
SERVER: new connection. chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56126]]
SERVER: connection readable. chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56126]]
Basic dXNlcjpwYXNzd29yZE5vdENoZWNrZWRBbnl3YXk=
SERVER: orderlyCloseChannel chan[java.nio.channels.SocketChannel[connected local=localhost/127.0.0.1:35395 remote=/127.0.0.1:56126]]
reading
Hello .
finished reading

java.lang.RuntimeException: Authenticator was called 0 times. Should be 1
    at AuthHeaderTest.except(AuthHeaderTest.java:117)
    at AuthHeaderTest.main(AuthHeaderTest.java:110)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
    at java.lang.Thread.run(Thread.java:821)

I am going to assume the plan is to:

  • look at history through TRSS to see if failing is only happening on a particular machine or group of machines
  • grind these intermittent tests to determine frequency of failure for each
  • look at each test with more scrutiny to determine if there are upstream changes that can be suggested to make them less flaky

Pretty much yeah, it's so that we can determine between actual intermittent failures and tests that just fail on a whim. Once we've decided it's an actual issue, we can investigate as you stated above

sun/nio/cs/FindDecoderBugs.java jdk11-j9-s390x-linux 19-Dec-2019

01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ed"[0/5] => null ""[0/2]
01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ed"[0/5] => null ""[0/2] (direct)
01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ee"[0/5] => null ""[0/19]
01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ee"[0/5] => null ""[0/19] (direct)
01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ef"[0/5] => null ""[0/19]
01:45:05  Unexpected exception: x-JISAutoDetect NullPointerException "1b 28 42 a4 ef"[0/5] => null ""[0/19] (direct)
01:45:05  Too many failures for x-JISAutoDetect

https://github.com/AdoptOpenJDK/openjdk-tests/issues/1511#issuecomment-566062160

java/io/RandomAccessFile/OpenSync.java jdk11-j9-s390x-linux 12-Dec-2019

java/io/RandomAccessFile/OpenSync.java jdk13-j9_x86-64-linux 6-Jan-2020 (Count 2)

@smlambert for #821 and #1259 I would agree as they are also collections of failing tests, I can already see some duplicates. The others are more specific issues about a single test so shouldn't we leave those alone? In this issue we're just collecting more information about the test and then creating its own separate issue when we can determine what the problem seems to be (eg. openj9, infra etc).

java/lang/ProcessHandle/TreeTest.java jdk13-j9-x64-windows-xl 8-Jan-2020

test TreeTest.test5(): failure
20:20:27  java.lang.AssertionError: expected at least 14, actual: 11 expected [true] but found [false]
20:20:27    at org.testng.Assert.fail(Assert.java:94)
20:20:27    at org.testng.Assert.failNotEquals(Assert.java:496)
20:20:27    at org.testng.Assert.assertTrue(Assert.java:42)
20:20:27    at TreeTest.test5(TreeTest.java:451)
20:20:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20:20:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20:20:27    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20:20:27    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
20:20:27    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
20:20:27    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
20:20:27    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:821)
20:20:27    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1131)
20:20:27    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
20:20:27    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
20:20:27    at org.testng.TestRunner.privateRun(TestRunner.java:773)
20:20:27    at org.testng.TestRunner.run(TestRunner.java:623)
20:20:27    at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
20:20:27    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
20:20:27    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
20:20:27    at org.testng.SuiteRunner.run(SuiteRunner.java:259)
20:20:27    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
20:20:27    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
20:20:27    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
20:20:27    at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
20:20:27    at org.testng.TestNG.run(TestNG.java:1018)
20:20:27    at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
20:20:27    at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:54)
20:20:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
20:20:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
20:20:27    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20:20:27    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
20:20:27    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
20:20:27    at java.base/java.lang.Thread.run(Thread.java:847)

java/security/KeyRep/Serial jdk8-j9-s390x-linuxXL 10-Jan-2020

java.lang.SecurityException: PBE test failed
    at Serial.main(Serial.java:168)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    at java.lang.Thread.run(Thread.java:821)

java/net/httpclient/DependentPromiseActionsTest.java
jdk13-j9-x64-linux 09-Jan-2020
jdk11-j9-x64-linux 12-Jan-2020

testng -Djdk.internal.httpclient.debug=true DependentPromiseActionsTest

20:37:40  test DependentPromiseActionsTest.testAsStringAsync("http://localhost:35296/http2/chunk/y", true, SemaphoreStaller): failure
20:37:40  java.lang.AssertionError
20:37:40    at DependentPromiseActionsTest.testDependent(DependentPromiseActionsTest.java:366)
20:37:40    at DependentPromiseActionsTest.testDependent(DependentPromiseActionsTest.java:336)
20:37:40    at DependentPromiseActionsTest.testAsStringAsync(DependentPromiseActionsTest.java:295)

java/lang/invoke/RevealDirectTest.java jdk8-j9-x64-windows 12-Jan-2020
RAISED: https://github.com/eclipse/openj9/issues/8268

08:12:32  *** FAIL: public final native java.lang.invoke.MethodHandle.invoke(Object[])Object => java.lang.AssertionError: expected:<4369> but was:<401>
08:12:32  java.lang.AssertionError: expected:<4369> but was:<401>
08:12:32    at org.junit.Assert.fail(Assert.java:93)
08:12:32    at org.junit.Assert.failNotEquals(Assert.java:647)
08:12:32    at org.junit.Assert.assertEquals(Assert.java:128)
08:12:32    at org.junit.Assert.assertEquals(Assert.java:472)
08:12:32    at org.junit.Assert.assertEquals(Assert.java:456)
08:12:32    at test.java.lang.invoke.RevealDirectTest.consistent(RevealDirectTest.java:639)
08:12:32    at test.java.lang.invoke.RevealDirectTest.testWithMember(RevealDirectTest.java:718)
08:12:32    at test.java.lang.invoke.RevealDirectTest.testOnMembersImpl(RevealDirectTest.java:384)
08:12:32    at test.java.lang.invoke.RevealDirectTest.testOnMembers(RevealDirectTest.java:357)
08:12:32    at test.java.lang.invoke.RevealDirectTest.testMethodHandleInvokes(RevealDirectTest.java:184)

java/util/concurrent/ScheduledThreadPoolExecutor/BasicCancelTest.java
jdk11-j9-x64-macXL 12-Jan-2020

22:19:14  Timeout information:
22:19:14  Running jstack on process 50361
22:19:14  2020-01-12T22:19:08.670622
22:19:14  Error getting data from 50361: Exception connecting to 50361

22:19:14  Passed = 13, failed = 0

java/util/Locale/LocaleProvidersRun.java jdk11-j9-x64-mac 13-Jan-2020

23:40:27  Unhandled exception
23:40:27  Type=Segmentation error vmState=0x0005ffff
23:40:27  J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000000

23:40:27  STDERR:
23:40:27  java.lang.RuntimeException: Unexpected exit code: 255
23:40:27    at LocaleProvidersRun.testRun(LocaleProvidersRun.java:172)
23:40:27    at LocaleProvidersRun.main(LocaleProvidersRun.java:93)
23:40:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
23:40:27    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23:40:27    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
23:40:27    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
23:40:27    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
23:40:27    at java.base/java.lang.Thread.run(Thread.java:834)

java/nio/file/WatchService/LotsOfCancels.java.LotsOfCancels

12:15:28 LotsOfCancels
12:15:28 STDERR:
12:15:28 java.lang.RuntimeException: Thread pool did not terminate
12:15:28 at LotsOfCancels.main(LotsOfCancels.java:67)
12:15:28 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:15:28 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:15:28 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:15:28 at java.lang.reflect.Method.invoke(Method.java:498)
12:15:28 at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
12:15:28 at java.lang.Thread.run(Thread.java:821)
12:15:28
12:15:28 JavaTest Message: Test threw exception: java.lang.RuntimeException
12:15:28 JavaTest Message: shutting down test
12:15:28

Platform
s390x_linux

Detail
sanity.openjdk
java/rmi/activation/rmidViaInheritedChannel/InheritedChannelNotServerSocket.java.InheritedChannelNotServerSocket

java.rmi.server.ExportException: Port already in use: 60004; nested exception is:
13:28:12 java.net.BindException: Address already in use (Bind failed)
13:28:12 at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
13:28:12 at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
13:28:12 at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
13:28:12 at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
13:28:12 at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:237)
13:28:12 at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:213)
13:28:12 at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:198)
13:28:12 at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:203)
13:28:12 at InheritedChannelNotServerSocket.main(InheritedChannelNotServerSocket.java:91)
13:28:12 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

java/util/TimeZone/TimeZoneTest.java Test_openjdk8_hs_sanity.openjdk_x86-64_linux 4-2-2020
Looks like a jenkins issue, test passes locally #1356

TimeZoneTest {
  TestShortZoneIDs {
    Fail: Expected Zone["BET", GMT-3:00, Standard]; got Zone["BET", GMT-3:00, Daylight]
    Uncaught exception thrown in test method TestShortZoneIDs

java.lang.RuntimeException: Uncaught exception thrown in test method TestShortZoneIDs
    at IntlTest.errImpl(IntlTest.java:190)
    at IntlTest.errln(IntlTest.java:177)
    at IntlTest.run(IntlTest.java:123)
    at TimeZoneTest.main(TimeZoneTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
    at java.lang.Thread.run(Thread.java:748)

New list for Feb - https://github.com/AdoptOpenJDK/openjdk-tests/issues/1596 can you copy that over @M-Davies ?

Was this page helpful?
0 / 5 - 0 ratings