Elasticsearch: Tests: LuceneTests are failing on Java9

[spinscale]

Seems that LuceneTests.testMMapHackSupported fails regularly on Java 9 right now, independent from the ES version

REPRODUCE WITH: gradle :core:test -Dtests.seed=ECC3BC902EFF7263 -Dtests.class=org.elasticsearch.common.lucene.LuceneTests -Dtests.method="testMMapHackSupported" -Dtests.security.manager=true -Dtests.jvm.argline="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED" -Dtests.locale=mas-KE -Dtests.timezone=Zulu

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.1+java9-periodic/540/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.0+java9-periodic/1324/console

[mikemccand]

Hmm, this is not good: it means on Java 9 we (ES) are failing to use Lucene's unmapping best efforts. Here's the root cause:

FAILURE 0.02s J0 | LuceneTests.testMMapHackSupported <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: MMapDirectory does not support unmapping: Unmapping is not supported on this platform, because internal Java APIs are not compatible to this Lucene version: java.lang.IllegalAccessException: symbolic reference class is not accessible: class jdk.internal.ref.Cleaner, from org.apache.lucene.store.MMapDirectory (unnamed module @77dd2cd6)
   >    at __randomizedtesting.SeedInfo.seed([ECC3BC902EFF7263:1D3CDF5BF2E129A4]:0)
   >    at org.elasticsearch.common.lucene.LuceneTests.testMMapHackSupported(LuceneTests.java:408)

I don't think it's a missing security permission, because Lucene throws a different message in that case.

Maybe our 1.9-ea is too old in the ES jenkins instance? We are running with 9-ea-150 but from https://jdk9.java.net/ it looks like it's up to ea-151.

Otherwise I'm not sure what's wrong ... @uschindler any pointers welcome :)

[jimczi]

I think this issue is fixed in Lucene for java9 150+:
https://issues.apache.org/jira/browse/LUCENE-6989
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7e03427 ?

I think there is also another issue that should prevent java9 to build:
https://issues.apache.org/jira/browse/LUCENE-7596

There is an ongoing discussion on the Lucene dev mailing list regarding the next release compatibility with java9.

[mikemccand]

Thanks for the pointers @jimczi!

I think this issue is fixed in Lucene for java9 150+

OK I think I understand now: Lucene 6.4.0 will have this fix (https://issues.apache.org/jira/browse/LUCENE-6989), to use unmapping with Java 9-ea 150+, but Lucene 6.3.0 (which ES 5.0, 5.1 are using) cannot do unmapping with Java 9-ea 150+.

So we should disable this test in ES 5.0, 5.1, but keep it in ES 5.x (5.2)? And notify ES users that if they use Java 9, they should use ES 5.2+ so unmap works.

[uschindler]

Hi,

first: If you have tests that fail on any Java version if unmapping does not work, it should ignore this failure. You should add some assumeTrue(MMapDirectory.UNMAP_NOT_SUPPORTED_REASON, MMapDirectory.UNMAP_SUPPORTED) to disable test exceution.

As Lucene tests only break on windows if unmapping is not supported, we have a special Assume in LuceneTestCase that enforces that unmapping works on windows only, for other OS it is happy also if unmapping does not work (it's just bad that it does not work for disk usage, but does not prevent tests from succeeding): LuceneTestCase.assumeWorkingMMapOnWindows()

second: Under some circumstances Lucene 6.0 - Lucene 6.3 and MMapDirectory completely fails, because the static initializer does not catch a new Java 9 RuntimeException (my fault, I am so sorry) and breaks class initialization (you don't see this in your tests, what Lucene version and which permissions are you using?). So people may completely fail unless they use another directory implementation.

Lucene 6.4 will be compatible with Java 9 b150, but it will no longer work with Java 9 build 100 to 149, because the code to support those versions was removed. It is very likely that the new variant will work also with the final release, because they added an API for unmapping in build 150 (although it is in sun.misc.Unsafe, but that is what it is: it's unsafe, because it may crush you JVM if you misuse the API). The new hack will of course work with Java 8 and very early Java 9 versions.

To fix the mmap issue in general, we have to wait for Java 10, there are some ideas of changing Hotspot and Java's memory model (a new type of field access with "volatile only on safepoint" semantics using VarHandles) that will be integrated internally into DirectByteBuffer. When you then "officially" unmap a bytebuffer it will also trigger a JVM safepoint, that makes a field in ByteBuffer suddenly behaving "volatile". After that it unmaps and sets the field to null, because its code that accesses the bytebuffer will do the volatile check. It's just too late for Java 9. Once this is done, ByteBuffer will be java.io.Closeable or similar and you can "officially" call ByteBuffer.close(), but until this happens we have to hack around it. I will discuss this with Andrew Haley and Mark Reinhold (+ Alan Bateman) on FOSDEM :-) I am sure the NETTY people will also happy about this!

I'd wait to make Java 9 compatibility public until Java 9 is out there :-) Once it is out we can offcially say: Use a minimum of that version of Elasticsearch using that version of Lucene. If it is urgent we may also backport the final fix to older Lucene versions and release bugfixes for 6.0, 6.1, 6.2, 6.3, so Elasticsearch can make the mainline versions (5.x) compatible to Java 9.

[uschindler]

I just noticed, Lucene 6.0 till 6.3 will not break fatally anymore with Java 9 build 150+. It just disables unmapping (as it should - which triggers your test failure). The reason is the Lucene-proposed Unsafe fix included in build 150. So my old (broken) code only breaks fatally in build 148 and 149, so all is fine! Users will be able to start ES, but unmapping just does not work.

[mikemccand]

Thanks @uschindler for the explanations! The ES test is here to catch us if unmap is unexpectedly not working, but in this case (Lucene < 6.3.0, Java 1.9 ea 150+) we know unmap does not work.

So I'll just open a PR to add an assume to this test, for just 5.0, 5.1, that if the java version is 1.9, then unmap won't work.

On 5.x we already use Lucene's 6.x snapshot (soon to be 6.4.0) which has your recent fixes for LUCENE-6989 (thank you!) so the test case doesn't need the assume there.

And I agree we should make no promises about Java 9 support until Java 9 is released!

[s1monw]

@uschindler for the rescue !!! thanks man

[mikemccand]

Fixed in 5.0 and 5.1.