Elasticsearch: [CI] Failures caused by Could not initialize class jdk.internal.misc.ExtendedMapMode

Created on 27 Dec 2019  路  11Comments  路  Source: elastic/elasticsearch

CorruptionUtilsTests.testLuceneCheckIndexIgnoresLast4Bytes fails with 


java.lang.InternalError: java.security.AccessControlException: access denied ("java.lang.reflect.ReflectPermission" "suppressAccessChecks") 
at __randomizedtesting.SeedInfo.seed([F5460B55BBCC2378:70578525AA44A9A7]:0)
at java.base/jdk.internal.misc.ExtendedMapMode.<clinit>(ExtendedMapMode.java:46)    
        at java.base/sun.nio.ch.FileChannelImpl.isSync(FileChannelImpl.java:1119)   
        at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:994)   
        at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121) 
        at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121) 
        at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121) 
        at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267)    
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242)  
        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2862)   
        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:747)    
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)  
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)  
        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)  
        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:287)   
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:846) 
        at org.elasticsearch.index.store.Store.newAppendingIndexWriter(Store.java:1541) 
        at org.elasticsearch.index.store.Store.associateIndexWithNewTranslog(Store.java:1448)   
        at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:416) 
        at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:94) 
        at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:285)    
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:92)  
        at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1841)  
        at org.elasticsearch.index.shard.IndexShardTestCase.recoverFromStore(IndexShardTestCase.java:799)   
        at org.elasticsearch.index.shard.IndexShardTestCase.recoverShardFromStore(IndexShardTestCase.java:550)  
        at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:512)    
        at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:499)    
        at org.elasticsearch.index.shard.IndexShardTestCase.newStartedShard(IndexShardTestCase.java:477)    
        at org.elasticsearch.test.CorruptionUtilsTests.testLuceneCheckIndexIgnoresLast4Bytes(CorruptionUtilsTests.java:45)

Reproduce on master with

./gradlew ':test:framework:test' --tests "org.elasticsearch.test.CorruptionUtilsTests.testLuceneCheckIndexIgnoresLast4Bytes" \
  -Dtests.seed=F5460B55BBCC2378 \
  -Dtests.security.manager=true \
  -Dtests.locale=lt-LT \
  -Dtests.timezone=America/Adak \
  -Dcompiler.java=13

Latest failure occurred on 6.8. Reproduce with 

./gradlew ':test:framework:unitTest' \
  -Dtests.seed=341C2A6EF508110D \
  -Dtests.class=org.elasticsearch.test.CorruptionUtilsTests \
  -Dtests.method="testLuceneCheckIndexIgnoresLast4Bytes" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-FJ \
  -Dtests.timezone=America/Mexico_City \
  -Dcompiler.java=12 \
  -Druntime.java=13

Build scans available:
[6.8] https://gradle-enterprise.elastic.co/s/drjfxhcoq5d3e
[7.5.2] https://gradle-enterprise.elastic.co/s/kvqlovsjbsa4i
[7.6.0] https://gradle-enterprise.elastic.co/s/boxheveulwx7a
[master] https://gradle-enterprise.elastic.co/s/ahghyu74kd4zo

I was unable to reproduce this locally, but presumably, we need to wrap a store operation withdoPrivileged

:DistributeStore Distributed jvm bug
Source
picture andreidan

Most helpful comment

Submitted a bug report to the JDK for this. I believe ExtendedMapMode should use doPrivileged in its static initialization.

picture henningandersen on 30 Dec 2019
👍3

All 11 comments

Pinging @elastic/es-distributed (:Distributed/Store)

elasticmachine picture elasticmachine on 27 Dec 2019

This is a java 14 only issue. It does reproduce with the EA java 14 build:

./gradlew ':test:framework:test' --tests "org.elasticsearch.test.CorruptionUtilsTests.testLuceneCheckIndexIgnoresLast4Bytes" -Dtests.seed=F5460B55BBCC2378 -Dtests.security.manager=true -Dtests.locale=lt-LT -Dtests.timezone=America/Adak -Dcompiler.java=13 -Druntime.java=14

Will investigate further, but wanted to note this here to avoid the tests being muted unnecessarily.

henningandersen picture henningandersen on 30 Dec 2019

Submitted a bug report to the JDK for this. I believe ExtendedMapMode should use doPrivileged in its static initialization.

henningandersen picture henningandersen on 30 Dec 2019
👍3

a few more: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/424/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk13,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/425/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=openjdk13,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/426/console

ebadyano picture ebadyano on 2 Jan 2020

This build uses ES_RUNTIME_JAVA=openjdk14so I think its related:

Log | https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/426/console
-- | --
Build Scans | https://gradle-enterprise.elastic.co/s/3lalkokmd4iey

Lots of failures like these in the logs:

Throwable #1: junit.framework.AssertionFailedError: Unexpected exception type, expected IndexShardRecoveryException but got java.lang.NoClassDefFoundError: Could not initialize class jdk.internal.misc.ExtendedMapMode
13:53:31    >   at __randomizedtesting.SeedInfo.seed([3D73D660E5C1FC35:8B003E96548B630F]:0)
13:53:31    >   at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2692)
13:53:31    >   at org.apache.lucene.util.LuceneTestCase.expectThrows(LuceneTestCase.java:2681)
13:53:31    >   at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandTests.testCleanWithCorruptionMarker(RemoveCorruptedShardDataCommandTests.java:362)
13:53:31    >   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
13:53:31    >   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
13:53:31    >   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:53:31    >   at java.base/java.lang.reflect.Method.invoke(Method.java:564)
13:53:31    >   at java.base/java.lang.Thread.run(Thread.java:832)
13:53:31    > Caused by: java.lang.NoClassDefFoundError: Could not initialize class jdk.internal.misc.ExtendedMapMode
13:53:31    >   at java.base/sun.nio.ch.FileChannelImpl.isSync(FileChannelImpl.java:1119)
13:53:31    >   at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:994)
cbuescher picture cbuescher on 3 Jan 2020

Different test same error:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.5+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=openjdk14,nodes=general-purpose/164/console
https://gradle-enterprise.elastic.co/s/abkec44l3eduw\

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=69, name=elasticsearch[node_s0][refresh][T#9], state=RUNNABLE, group=TGRP-CancelTests]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class jdk.internal.misc.ExtendedMapMode


Stack Trace
at __randomizedtesting.SeedInfo.seed([6D20A6EC6A8ADC9C]:0)
at java.base/sun.nio.ch.FileChannelImpl.isSync(FileChannelImpl.java:1119)
at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:994)
at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121)
at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121)
at org.apache.lucene.mockfile.FilterFileChannel.map(FilterFileChannel.java:121)
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:267)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:242)
at org.elasticsearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:141)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:100)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:89)
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5014)
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:574)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:513)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:555)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:722)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:494)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:297)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:272)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:165)
at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:66)
at org.elasticsearch.index.engine.ElasticsearchReaderManager.refreshIfNeeded(ElasticsearchReaderManager.java:40)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:339)
at org.elasticsearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:321)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1603)
at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1580)
at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:954)
at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.lambda$shardOperationOnPrimary$0(TransportShardRefreshAction.java:63)
at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:285)
at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:62)
at org.elasticsearch.action.admin.indices.refresh.TransportShardRefreshAction.shardOperationOnPrimary(TransportShardRefreshAction.java:41)
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:917)
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108)
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:394)
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$22(IndexShard.java:2796)
at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113)
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285)
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237)
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2770)
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:858)
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63)
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)

I updated the issue title to reflect the fact that the failure is not limited to a specific test

davidkyle picture davidkyle on 6 Jan 2020

Removing the test-failure label, since this is an issue that needs resolving before JDK 14 can be supported.

henningandersen picture henningandersen on 6 Jan 2020

The JDK bug report is here:

https://bugs.openjdk.java.net/browse/JDK-8236582

and is marked fixed January 9th. A build is not yet available, so will leave issue open for now.

henningandersen picture henningandersen on 10 Jan 2020
🎉1

This has been fixed in the latest jdk-14 build 32 that came out Wednesday.

henningandersen picture henningandersen on 17 Jan 2020

Our CI agents have been updated with the latest JDK 14 build.

mark-vieira picture mark-vieira on 22 Jan 2020

@henningandersen Can this be closed now?

ywelsch picture ywelsch on 23 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

matthughes picture matthughes  路  3Comments
malpani picture malpani  路  3Comments
abtpst picture abtpst  路  3Comments
ttaranov picture ttaranov  路  3Comments
DhairyashilBhosale picture DhairyashilBhosale  路  3Comments