Openjdk-tests: Build/test pipeline should use the same commits for each job

[adam-thorpe]

jdk/nio/zipfs/DirectoryStreamTests.java
jdk/nio/zipfs/InvalidZipHeaderTests.java
jdk/nio/zipfs/jarfs/JFSTester.java

Failing on JDK11 on almost all platforms, both hotspot and openj9

These three tests seem to failing due to a recently backported change from JDK13:
https://github.com/ibmruntimes/openj9-openjdk-jdk11/commit/d98b2eb726599a50b6355ee4ee8016d172a51f1a
They seem to have problems when reading jar files (interacting with META-INF). As far as I'm aware this problem doesn't seem to be persistent on JDK13 so am not sure the cause here.

eg.

test JFSTester.testWalk(): failure
java.lang.AssertionError: Sets differ: expected [This is leaf 2.
, This is leaf 1.
, This is version 9 leaf 5.
, This is version 9 leaf 4.
, This is version 9 leaf 3.
, This is version 9 leaf 6.
] but got [This is version 9 leaf 4.
, This is leaf 1.
, This is version 9 leaf 3.
, This is leaf 2.
]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.assertEquals(Assert.java:808)
    at org.testng.Assert.assertEquals(Assert.java:786)
    at JFSTester.testWalk(JFSTester.java:110)

[sophia-guo]

Seems like those failures happened recently https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_hs_sanity.openjdk_x86-64_linux/. May not be related with the seven months ago change ibmruntimes/openj9-openjdk-jdk11@d98b2eb.

Test_upstream is fine https://ci.adoptopenjdk.net/view/Test_upstream/job/Test_openjdk11_hs_sanity.system_x86-64_linux_upstream/ though may not be the latest. If latest test_upstream have similar issues we can open the issue in JBS. @jerboaa

[adam-thorpe]

@sophia-guo because this change was created in jdk next seven months ago and then backported to jdk11, its marked as the same date (both in mercurial and github).
Merged in openj9 2 days ago: https://github.com/ibmruntimes/openj9-openjdk-jdk11/commit/05fdebd425ee9962c8607daf932f4254503ab6bd#diff-a78bc512ab9028e23111986846fd86bd

[M-Davies]

That change may also be why https://github.com/AdoptOpenJDK/openjdk-tests/issues/1441 is failing too.

[adam-thorpe]

Okay, think I have found the root of the problem which is why we had a large number of test failures on all platforms yesterday but they all fixed themselves overnight. Taking openj9 mac as an example, 11th novembers build used a pre-merge commit:

From https://github.com/ibmruntimes/openj9-openjdk-jdk11
   e7da16be04..5961cb4f17  openj9         -> origin/openj9
   cc4272e86c..ac999a4af3  openj9-staging -> origin/openj9-staging
HEAD is now at 5961cb4f17 Merge pull request #226 from theresa-m/fix_7332_invoke

but then the resulting sanity openjdk test run used the commit after the merge:

20:53:42       [exec]   Fetch URL: https://github.com/ibmruntimes/openj9-openjdk-jdk11.git
20:53:42       [exec] sha:
20:53:42       [exec] 3cc03bc6fbbab6bbd54eecb860e007f44e14656e

Both the tests and the source code was changed in the merge, so we were running the old source against the newer tests.

[sophia-guo]

https://github.com/AdoptOpenJDK/openjdk-tests/issues/1440#issuecomment-553338131 that happens if there is change between the gap of sdk build and test build. Ideally build can pass in the sha to avoid this. https://github.com/AdoptOpenJDK/openjdk-tests/issues/739#issuecomment-446651375 We may need to open an issue in build repo too.

[sophia-guo]

Those tests still fail with hotspot. https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_hs_sanity.openjdk_x86-64_linux/83/

Tests using default master with sha 3d43ca645bcfc5f4f6796a561553aaad6c0edaab

20:29:10       [exec] ================================================
20:29:10       [exec] timestamp: 20191114-192907
20:29:10       [exec] repo dir: openjdk-jdk
20:29:10       [exec] git repo: 
20:29:10       [exec]   Fetch URL: https://github.com/AdoptOpenJDK/openjdk-jdk11u.git
20:29:10       [exec] sha:
20:29:10       [exec] 3d43ca645bcfc5f4f6796a561553aaad6c0edaab

while source is using different sha https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-linux-x64-hotspot/338/consoleFull

origin  https://github.com/adoptopenjdk/openjdk-jdk11u.git (fetch)
origin  https://github.com/adoptopenjdk/openjdk-jdk11u.git (push)
jdk11
origin  https://github.com/adoptopenjdk/openjdk-jdk11u.git (fetch)
Resetting the git openjdk source repository at /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-linux-x64-hotspot/workspace/build/src in 10 seconds...
Pulling latest changes from git openjdk source repository
From https://github.com/adoptopenjdk/openjdk-jdk11u
   6e76ed16e2..3d43ca645b  master       -> origin/master
 * [new tag]               jdk-11.0.6+3 -> jdk-11.0.6+3
Fetching origin
HEAD is now at 327194045b Merging latest patches from release branch

Looks like in AdoptOpenJDK nightly hotspot build is using dev instead of 'master' branch of https://github.com/AdoptOpenJDK/openjdk-jdk11u .

@karianna Could you confirm if this is on purpose?

[smlambert]

PR #1447 and #1448 should have resolved this, please confirm.

[adam-thorpe]

We won't really know if this works until we get the unlikely scenario of a build being started, then a merge come through from openjdk before the tests are ran. I guess it's safe to assume that this does work and then if we see the same problem occur again then to update this thread.