Druid: Kafka Indexing breaks due to Mismatched DatSourceMetadata types during InternalReset call

Created on 19 Jun 2019 · 14Comments · Source: apache/druid

Affected Version

0.14.0, 0.14.2, 0.15.0-incubating

Description

This is pretty hard to reproduce as it seems to be a race condition during unexpected termination of the middle manager process.

Something happens, I suspect when middle manager shuts down unexpectedly at a specific yet not currently discovered time, that causes the datasource metadata record to have the wrong type for the sequence stream numbers. When middle manger comes back up and is creating the new indexing tasks, it will fail with the following stack trace:

aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | 2019-06-10T14:02:32,642 ERROR [KafkaSupervisor-npav-ts-metrics] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - SeekableStreamSupervisor[npav-ts-metrics] failed to handle notice: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor, exceptionType=class org.apache.druid.java.util.common.IAE, exceptionMessage=Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers, noticeClass=RunNotice}
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | org.apache.druid.java.util.common.IAE: Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers.minus(SeekableStreamEndSequenceNumbers.java:159) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.SeekableStreamDataSourceMetadata.minus(SeekableStreamDataSourceMetadata.java:95) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.resetInternal(SeekableStreamSupervisor.java:1147) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2380) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2357) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2254) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1013) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:265) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:724) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_66-internal]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_66-internal]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66-internal]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66-internal]
aod_overlord.1.dcc7ttv9r52t@datahub2-leader-d0xb | at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]

The issue is that the minus operation requires a SeekableStreamSequenceNumber of the same implementation (start or end) to be successful, but the currentMetadata and resetMetadata values are of different types.

I tried to correct this issue via conversion of the currentMetadata and resetMetadata both to sequence start numbers via the existing helper method 'asStartMetadata' on the SeekableStreamEndSequenceNumber class, but this caused further conversion issues on subsequent indexing tasks as documented in this PR: https://github.com/apache/incubator-druid/pull/7831

After trying some other solutions, it became clear to me that I do not know enough about the way the start and end numbers were being modified to inject a patch using the built in conversion function. Since we are only using Kafka ingestion (not Kinesis), I re-evaluated the SeekableStreamEndSequenceNumber and SeekableStreamStartSequenceNumber classes and realized that they are pretty much the same thing and that only the "exclusivePartitions' property of the SeekableStreamStartSequenceNumber is used for kinesis streaming. So it seems like both classes are not really required as one class could provide the functionality needed.

I believe this entire issue could be resolved by consolidating those two classes into one class as I am not certain two separate classes are required to manage the partition offset states for the start and end offsets independently.

Current Workaround

I have a custom build where I have commented out the SeekableStreamEndSequenceNumber class and replaced all instances of SeekableStreamEndSequenceNumber with SeekableStreamStartSequenceNumber. Anytime the constructor for SeekableStreamStartSequenceNumber is called, I am using Collections.emptySet() for the third parameter for the exclusivePartitionMap as Kafka indexing does not use this. So far this workaround has "corrected" the bad datasource metadata and indexing tasks are all passing successfully. I am currently testing failure scenarios to try and reproduce the original race condition bug to validate this is working.

Area - Streaming Ingestion Bug

Source

acdn-mpreston

All 14 comments

@acdn-mpreston thank you for digging into this issue. resetInternal can be called in several situations and I think that's why this error occurs occasionally.

I think the bug is, the dataSourceMetadata stored in metadata store is always the "end" metadata, but some places create "start" metadata for reset. Here is the list of where it creates start metadata.

I think changing these places to create end metadata instead would fix this issue.

jihoonson on 19 Jun 2019

👍1

This issue won't be fixed in 0.15.0 since it's not a regression.

jihoonson on 19 Jun 2019

Fantastic @jihoonson, thanks for the update. I will make another custom build with the changes you have suggested and see how it behaves in our test environment.

acdn-mpreston on 19 Jun 2019

@acdn-mpreston awesome. Thank you!

jihoonson on 20 Jun 2019

@jihoonson I made the changes you have suggested but have run into issues further down the ingestion pipeline. Looks like when forcing those 3 locations to use End numbers instead of Start numbers, there is still another location that is creating the Start numbers as per the error I am including below.

It was running into these types of additional spots that required special handling that started me down the path that maybe consolidating the two classes into one would save some trouble given that they seem to be providing almost the same functionality:

2019-06-20T14:41:14,535 ERROR [KafkaSupervisor-npav-ts-metrics-15m] org.apache.druid.metadata.SQLMetadataStorageActionHandler - aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb | com.fasterxml.jackson.databind.JsonMappingException: Class org.apache.druid.indexi aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb aod_overlord.1.rmwhy8xqubgz@datahub2-leader-d0xb Encountered exception while deserializing task payload, setting task to null
ng.seekablestream.SeekableStreamStartSequenceNumbers is not assignable to org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers (through reference chain: org.apache.druid.indexing.kafka.KafkaIndexTask["ioConfig"]->org.apache.druid.indexing.kafka.KafkaIndexTaskIOConfig["startPartitions"])
| at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:177) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1474) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:378) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1099) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:296) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:122) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:93) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:992) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:518) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:463) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:378) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1099) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:296) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:136) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:122) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:93) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:131) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:42) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2836) ~[jackson-databind-2.6.7.jar:2.6.7]
| at org.apache.druid.metadata.SQLMetadataStorageActionHandler$TaskInfoMapper.map(SQLMetadataStorageActionHandler.java:355) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.metadata.SQLMetadataStorageActionHandler$TaskInfoMapper.map(SQLMetadataStorageActionHandler.java:334) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.skife.jdbi.v2.Query$4.munge(Query.java:183) ~[jdbi-2.63.1.jar:2.63.1]
| at org.skife.jdbi.v2.QueryResultSetMunger.munge(QueryResultSetMunger.java:43) ~[jdbi-2.63.1.jar:2.63.1]
| at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1340) ~[jdbi-2.63.1.jar:2.63.1]
| at org.skife.jdbi.v2.Query.fold(Query.java:173) ~[jdbi-2.63.1.jar:2.63.1]
| at org.skife.jdbi.v2.Query.list(Query.java:82) ~[jdbi-2.63.1.jar:2.63.1]
| at org.skife.jdbi.v2.Query.list(Query.java:75) ~[jdbi-2.63.1.jar:2.63.1]
| at org.apache.druid.metadata.SQLMetadataStorageActionHandler.lambda$getActiveTaskInfo$4(SQLMetadataStorageActionHandler.java:296) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
| at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:133) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[druid-core-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114) ~[druid-core-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104) ~[druid-core-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:133) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:142) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.metadata.SQLMetadataStorageActionHandler.getActiveTaskInfo(SQLMetadataStorageActionHandler.java:290) ~[druid-server-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.indexing.overlord.MetadataTaskStorage.getActiveTasks(MetadataTaskStorage.java:195) ~[druid-indexing-service-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.discoverTasks(SeekableStreamSupervisor.java:1233) ~[druid-indexing-service-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1004) ~[druid-indexing-service-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:265) ~[druid-indexing-service-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:724) ~[druid-indexing-service-0.14.1-incubating-SNAPSHOT.jar:0.14.1-incubating-SNAPSHOT]
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_66-internal]
| at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_66-internal]
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66-internal]
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66-internal]
| at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
| Caused by: java.lang.IllegalArgumentException: Class org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers is not assignable to org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers
| at com.fasterxml.jackson.databind.JavaType._assertSubclass(JavaType.java:466) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.JavaType.narrowBy(JavaType.java:149) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.type.TypeFactory.constructSpecializedType(TypeFactory.java:315) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:177) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:108) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:93) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:992) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:518) ~[jackson-databind-2.6.7.jar:2.6.7]
| at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:463) ~[jackson-databind-2.6.7.jar:2.6.7]
| ... 48 more

acdn-mpreston on 20 Jun 2019

Also, the segments fail to publish, even though the indexing task does complete work, with these changes made.

acdn-mpreston on 20 Jun 2019

👍1

Also, the segments fail to publish, even though the indexing task does complete work, with these changes made.

@acdn-mpreston @jihoonson
I have encountered this problem, can't avoid this problem by modifying the configuration? 0.14.1

w3z4y4 on 1 Jul 2019

I have this problem after modifying the supervisor task, it is very difficult to reproduce.

w3z4y4 on 1 Jul 2019

@jihoonson just letting you know that I have tried 0.15.0-incubating and can confirm that this issue does exist there as well.

acdn-mpreston on 3 Jul 2019

I faced this issue on 0.15.0-incubating. Not exactly sure what triggered it, but I did offset resets for some supervisors and restarted the middle manager around the time when it happened. However, I have not yet been able to reproduce it.

@acdn-mpreston any progress with the workaround, or any hints on how you managed to reproduce it?

gvsmirnov on 22 Jul 2019

@gvsmirnov I am still unclear as to the root cause of the issue, but the way I most commonly got in this state was during an upgrade of a system using druid 0.12.X to druid 0.14.X.

I do not have any workaround that is PR worthy, but if you only need Kafka ingestion (and not Kinesis) then my workaround is documented in the "Current Workaround" section of the issue description. The basic idea is to get rid of usage of the SeekableStreamEndSequenceNumber class and only use SeekableStreamStartSequenceNumber instances instead. This avoids all casting issues but it will break Kinesis streaming.

acdn-mpreston on 22 Jul 2019

@acdn-mpreston thanks for getting back to me. To clarify my case, I got this issue in a clean deployment of druid 0.15.0-incubating. It was not upgrading from any previous version, just a fresh installation.

I am indeed using only kafka, and will try the same workaround. Was a bit confused by your follow-up comment stating that you have run into issues further down the ingestion pipeline. Did you find the root cause of that one? Thanks.

gvsmirnov on 22 Jul 2019

@gvsmirnov yes, the issue can happen on a clean install as well, the upgrade path was just the easiest way I found to reproduce. :)

My comment of running into issues "further down the pipeline" refers to a fix I tried originally in a referenced PR. Unfortunately, that attempt failed after the supervisor reset itself as it was expecting an offset stored as a SeekableStreamEndSequenceNumber (as the reset metadata should always have offsets stored using this class) but in my PR I had cast it to a SeekableStreamStartSequenceNumber. This resulted in another class mismatch, this time on the "plus" method call.

This is what made me change my approach to just using SeekableStreamStartSequenceNumber class instances for storing the offset metadata as I clearly did not understand well enough all the places where the offset metadata would be retrieved and manipulated. By using only SeekableStreamEndSequenceNumber, there will never be a class mismatch.

acdn-mpreston on 22 Jul 2019

👍1

Hi @acdn-mpreston, just raised https://github.com/apache/incubator-druid/pull/8305. Hopefully it fixes this issue.

Looking at the stack trace you posted above, it looks like some old tasks have a wrong type of sequence number. Probably it would work if you delete those tasks from the metadata store manually.

It was running into these types of additional spots that required special handling that started me down the path that maybe consolidating the two classes into one would save some trouble given that they seem to be providing almost the same functionality:

It was a single class before but has split into two classes since it was too easy to make mistakes. Several bugs have fixed in https://github.com/apache/incubator-druid/pull/7291, https://github.com/apache/incubator-druid/pull/7332, and https://github.com/apache/incubator-druid/pull/7339.

jihoonson on 15 Aug 2019

Was this page helpful?

0 / 5 - 0 ratings