Openjdk-tests: Triage builds of JDK8/HotSpot arm32 with JIT enabled

[sxa]

Tests of sanity.openjdk via Grinder - need to find some time to triage as the JIT-enabled version will, by default, get released this week as 8u242. Happy for this to be moved somewhere else if anyone objects :-)

First (JIT) run: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1672/
Second run (no JIT, like last GA - timed out after 10h): https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1687/
Third (JIT) run https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1723/

non-JIT build: https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_arm_linux_hotspot_8u232b09.tar.gz
JIT build: https://api.adoptopenjdk.net/v2/binary/nightly/openjdk8?openjdk_impl=hotspot&os=linux&arch=arm&release=latest&type=jdk&heap_size=normal

[M-Davies]

@sxa555

Triage

1st Run

• com/sun/net/httpserver/SelCacheTest.java.SelCacheTest
  
  
  
  • Initially reported in (https://github.com/AdoptOpenJDK/openjdk-tests/issues/802) for JDK11-openj9. Also reported to be intermittent (https://bugs.openjdk.java.net/browse/JDK-8224603).
  
  
• java/net/Socket/HttpProxy.java.HttpProxy
  
  
  
  • Reported in (https://bugs.openjdk.java.net/browse/JDK-8223465). New Failure.
  
  
• java/net/URL/OpenStream.java.OpenStream
  
  
  
  • Excluded for JDK11-hs and JDK13-hs. New Failure.
  
  
• java/net/ipv6tests/TcpTest.java.TcpTest
  
  
  
  • Unexcluded recently (https://github.com/AdoptOpenJDK/openjdk-tests/pull/1517). Seems to be intermittent (https://github.com/AdoptOpenJDK/openjdk-tests/issues/1259). New Failure.
  
  
• java/nio/channels/SocketChannel/ExceptionTranslation.java.ExceptionTranslation
  
  
  
  • RAISED (https://github.com/AdoptOpenJDK/openjdk-tests/issues/340) for JDK9-hs with same failure. Also RAISED for JDK8-hs but never resolved, presumably because it started passsing again (https://github.com/AdoptOpenJDK/openjdk-tests/issues/827#issuecomment-456973398). Unexcluded in (https://github.com/AdoptOpenJDK/openjdk-tests/pull/1475)
  
  
• java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java.MapOpTest
  
  
  
  • No relevant history for the failing case. New Failure.
  
  
• java/util/stream/test/org/openjdk/tests/java/util/stream/SliceOpTest.java.SliceOpTest
  
  
  
  • Excluded for JDK8-hs/j9 (https://github.com/AdoptOpenJDK/openjdk-tests/blob/d2f0946813daaf3cc9708eae88072953579ab261/openjdk/ProblemList_openjdk8.txt#L250) on zlinux for timeout reason (not an ArrayOutOfBoundsException like it is here). New Failure.
  
  
• java/util/stream/test/org/openjdk/tests/java/util/stream/TabulatorsTest.java.TabulatorsTest
  
  
  
  • No relevant history for the failing case. New Failure.
  
  

2nd run

• Timeout after 10 hours. Caused by sun/nio/cs/OLD/TestIBMDB.java. Test is excluded in JDK8 for zlinux and all platforms for JDK9. Known to cause timeouts: https://github.com/AdoptOpenJDK/openjdk-tests/issues/1051

3rd run

• com/sun/net/httpserver/Test1.java.Test1
  
  
  
  • Excluded for JDK8-hs-ppc64 (https://github.com/AdoptOpenJDK/openjdk-tests/issues/827) for timeout reasons. Unexcluded in (https://github.com/AdoptOpenJDK/openjdk-tests/pull/1323). Fails in this instance with a SIGSEGV when running attempting to get the instance of a Cipher. New Failure.
  
  
• java/net/URL/OpenStream.java.OpenStream
  
  
  
  • See 1st run
  
  
• java/net/ipv6tests/TcpTest.java.TcpTest
  
  
  
  • See 1st run
  
  
• java/nio/channels/SocketChannel/ExceptionTranslation.java.ExceptionTranslation
  
  
  
  • See 1st run
  
  
• java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java.MapOpTest
  
  
  
  • See 1st run
  
  
• java/util/stream/test/org/openjdk/tests/java/util/stream/StreamLinkTest.java.StreamLinkTest
  
  
  
  • No relevant history for the failing case. New Failure.
  
  

A lot of the tests are failing with ArrayOutOfBoundsException that have not been seen before.

[sxa]

OpenStream is an issue with it resolving hostnames it shouldn't (Same as https://github.com/AdoptOpenJDK/openjdk-infrastructure/issues/984 - now resolved on the machine - re-running with JIT at https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1757/console)

TcpTest is giving a SocketException which doesn't quite indicate that it's the same failure as the references above

Am re-running the non-JIT version with (hopefully) IBMDB excluded at https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1751 and it's at 7 hours so we'll see if it gets to the end and whether the same failures occur. It doesn't seem to have shown any output for TcpTest yet

[M-Davies]

OpenStream is an issue with it resolving hostnames it shouldn't (Same as AdoptOpenJDK/openjdk-infrastructure#984 - now resolved on the machine - re-running with JIT at https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1757/console)

OpenStream passed. DistinctOpTest failed with a similar OutOfBoundsException to SliceOpTest and MapOpTest. No known history in infrastructure, build, jbs or tests repo.

TcpTest is giving a SocketException which doesn't quite indicate that it's the same failure as the references above

Tcp failed for https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1757/console too with the same exception. Might be worth waiting until the non-jit run completes to check if it is a JIT issue.

Am re-running the non-JIT version with (hopefully) IBMDB excluded at https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1751 and it's at 7 hours so we'll see if it gets to the end and whether the same failures occur. It doesn't seem to have shown any output for TcpTest yet

Looks like it didn't exclude the test for some reason 😕. Have re-run the job excluding the test from all linux distros https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1765/console. looking at the backlog for that machine, I see it might take while. https://ci.adoptopenjdk.net/job/Grinder/1758/console has hung due to FindDecoderBugs

[sxa]

Just kicked off a JIT-enabled run at https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1766 on another machine (Ubuntu) to see if it shows the same results as my Pi

[sxa]

More links for reference - re-runs of individual sections of sanity.openjdk on the non-JIT GA version to avoid the 10 hour timeout (Only nio has finished at the time of writing this - rest are queued up

• jdk_nio_0
• jdk_security1_0
• jdk_util_0
• jdk_rmi_0
• jdk_net_0

[M-Davies]

jdk_nio_0

- java/nio/MappedByteBuffer/Truncate.java.Truncate Excluded on zlinux
- java/nio/channels/FileChannel/Transfers.java.Transfers Previously known to cause timeouts
- sun/nio/cs/FindDecoderBugs.java.FindDecoderBugs Might be intermittent (#1511 and M2 triage)
- sun/nio/cs/OLD/TestIBMDB.java.TestIBMDB As discussed above
- sun/nio/cs/TestStringCoding.java.TestStringCoding Excluded for all platforms on openj9 initially as an unrelated intermittent problem. Not clear what the error was here. Timeout would have been flagged up? Exit code 143 is a SIGTERM command, needs more investigation
- sun/nio/cs/TestStringCodingUTF8.java.TestStringCodingUTF8 Excluded for all platforms on openj9 initially as an unrelated intermittent problem. New failure.

jdk_security1_0

- java/security/KeyPairGenerator/SolarisShortDSA.java.SolarisShortDSA Excluded on zlinux.
- java/security/KeyStore/PBETest.java.PBETest No timeout history, new failure
- java/security/MessageDigest/TestDigestIOStream.java.TestDigestIOStream Excluded on zlinux
- java/security/MessageDigest/TestSameLength.java.TestSameLength No timeout history, new failure
- java/security/Security/SynchronizedAccess.java.SynchronizedAccess Related Thread Cleanup Error reported upstream
- java/security/SignedObject/Chain.java.Chain No timeout history, new failure

jdk_util_0

*- Timed out after multiple tests timed out *

jdk_rmi_0

PASSED

jdk_net_0

- com/sun/net/httpserver/Test9a.java.Test9a Known to fail on jdk11-arm32
- java/net/URLPermission/URLTest.java.URLTest Known to fail on jdk11-arm32. Looks like a timeout.
- java/net/URL/OpenStream.java.OpenStream (10) All same error.
- java/net/ipv6tests/TcpTest.java.TcpTest (5) 1 reported as passed, 4 fails (all crash with the same error).

[karianna]

@smlambert - If we choose to accept these failures as 'intermittent or expected on arm32' what would the next logical step be? I assume we'd exclude these tests to get a green run and then investigate each more deeply?

[smlambert]

Yes, to proceed, we should exclude failing tests on this platform (with links to issues to track) so that we can run green, investigate/fix issues as aggressively as we can.

[karianna]

jdk_nio_0

- java/nio/MappedByteBuffer/Truncate.java.Truncate Excluded on zlinux
- java/nio/channels/FileChannel/Transfers.java.Transfers Previously known to cause timeouts
- sun/nio/cs/FindDecoderBugs.java.FindDecoderBugs Might be intermittent (#1511 and M2 triage)
- sun/nio/cs/OLD/TestIBMDB.java.TestIBMDB As discussed above
- sun/nio/cs/TestStringCoding.java.TestStringCoding Excluded for all platforms on openj9 initially as an unrelated intermittent problem. Not clear what the error was here. Timeout would have been flagged up? Exit code 143 is a SIGTERM command, needs more investigation
- sun/nio/cs/TestStringCodingUTF8.java.TestStringCodingUTF8 Excluded for all platforms on openj9 initially as an unrelated intermittent problem. New failure.

jdk_security1_0

- java/security/KeyPairGenerator/SolarisShortDSA.java.SolarisShortDSA Excluded on zlinux.
- java/security/KeyStore/PBETest.java.PBETest No timeout history, new failure
- java/security/MessageDigest/TestDigestIOStream.java.TestDigestIOStream Excluded on zlinux
- java/security/MessageDigest/TestSameLength.java.TestSameLength No timeout history, new failure
- java/security/Security/SynchronizedAccess.java.SynchronizedAccess Related Thread Cleanup Error reported upstream
- java/security/SignedObject/Chain.java.Chain No timeout history, new failure

jdk_util_0

*- Timed out after multiple tests timed out *

jdk_rmi_0

PASSED

jdk_net_0

- com/sun/net/httpserver/Test9a.java.Test9a Known to fail on jdk11-arm32
- java/net/URLPermission/URLTest.java.URLTest Known to fail on jdk11-arm32. Looks like a timeout.
- java/net/URL/OpenStream.java.OpenStream (10) All same error.
- java/net/ipv6tests/TcpTest.java.TcpTest (5) 1 reported as passed, 4 fails (all crash with the same error).

@M-Davies - Are you able to add these to the appropriate exclude file and reference this issue? I'm also happy to learn by looking over your shoulder when you do it :-)

[M-Davies]

@karianna Sure 😄. I'm basically going to be following the documentation at https://github.com/AdoptOpenJDK/openjdk-tests/tree/master/openjdk#exclude-a-testcase.

1) ~Creating issues for each of the failing tests (except for ones that already have relevant issues or JBS entries)~ See references above
2) Add each of these issues into ProblemList_openjdk8.txtspecifying aarch32 as the culprit. I.e.

java/net/ipv6tests/TcpTest.java https://github.com/AdoptOpenJDK/openjdk-tests/issues/1639    linux-aarch32

This will exclude these tests from testing on arm32
3) ~Create a PR to merge my problem list entries into master #1640~ MERGED

[karianna]

@M-Davies - Thank you! I appreciate you doing the heavy lifting on my behalf there, I owe you one!

[karianna]

OK, so the next step is to put this through a release again :-). @sxa555 - I might lean on you here for some advice. Are the ARM builds now set to the JIT'd implementation or Zero Impl?

[sxa]

JIT by default since https://github.com/AdoptOpenJDK/openjdk-build/commit/333235e6f4e2f24c6706567446b7e23e9b91061f went in. The nightlies come out with e.g.:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-202002210344-b02)
OpenJDK Client VM (AdoptOpenJDK)(build 25.252-b02, mixed mode)

I've scheduled another run at https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/880 from the jdk8u242-b08 tag which I believe is the correct one to use for arm32 release builds.

[smlambert]

I did not get to seeing the results of the referenced arm32 pipeline in time.

[sxa]

@smlambert Doesn't matter - it failed as the repo doesn't appear to have mirrored correctly :-(

[smlambert]

Egad, okay, I will leave this issue open until we can actually execute on it.

[karianna]

Repo is mirrored once more. We just need to get faster hardware to run the JIT tests.

[sxa]

@karianna Does that mean we could now re-release a .1 of 8u242 with JIT enabled?

[karianna]

I think we need to get better hardware first @gdams is working on it with AWS

[gdams]

looking at the aarch32 mirror we only have jdk8u242-b06 rather than jdk8u242-b08. Let's talk to Red Hat

[karianna]

@jerboaa - Is there a particular maintainer we should talk to?

[jerboaa]

@karianna We don't maintain that tree, AFAIK.

http://openjdk.java.net/census#aarch32-port

You are talking about this repository, right?
https://hg.openjdk.java.net/aarch32-port/jdk8u/ It has a jdk8u242-ga-aarch32-200120 tag which should be what you're looking for no?

[karianna]

@jerboaa Oh, it's Ed Nevill from ARM right? Let me go talk to him separately then :-). you're right on the repo

[smlambert]

I've rerun all of the tests that were thought to be previously excluded. The excludes were not effective (using linux-aarch32 instead of linux-arm in the ProblemList files, so no tests were actually excluded).

The test reruns show that tests are passing. If there are machines where they can not complete in a certain period of time and then fail due to timeout, I recommend removing those machines from the test farm.

I will close this. We can raise a fresh issue should there be other issues spotted.

[sxa]

@ShelleyLambert Which ones did you have to change from linux_aarch32 to linux_arm? Looking at openjdk-tests it looks like linux_arm stuff is only excluded in the OpenJ9 files (which isn't built on arm32) (And the referenced issue in that exclude file is closed)

[smlambert]

jtreg from openjdk project only knows to exclude stuff named linux-arm

if you look at the comments I added to any of the related issues that tests were excluded under, you will see what I am referring to (entries in ProblemLists that are then read by jtreg framework, https://github.com/AdoptOpenJDK/openjdk-tests/pull/2018).

[smlambert]

linux_arm entries will also be ignored (note '-' vs '_'), so I need to update my #2018 PR once I check the state of those tests (excluded under #440 )

[smlambert]

Also I should mention that @smlambert is my github ID, I do not see @ShelleyLambert mentions as that id was auto-created for me by my former employer whose email I no longer have access to.