Materialize: Support reading from Kinesis stream as of a timestamp/sequence number

Created on 12 Mar 2020  路  5Comments  路  Source: MaterializeInc/materialize

The current implementation of Kinesis sources will read in the entire stream upon connection. We should give users the option to create a source that starts at a specific:

  • sequence number: unique record identifier in a Kinesis stream (per shard and partition-key)
  • timestamp

Options are outlined more fully here: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html

A-integration C-feature docs-impact

All 5 comments

A quick thing to think out here: if the stream contains changes then it might be a mis-feature to start only part of the way through the stream. On the other hand, if the stream is just lines of text that we are either appending or windowing or something, then no real harm.

Just a heads up that one prospect gave me the valuable feedback that this feature is super important because _oftentimes the start of the stream is full of bad dev data_ and things only iron out (in terms of schema, columns being present, etc.) after some point in time. They know the point in time, but they do need the knob to _throw away data_ before that point in time. This was in reference to their Kafka installation (related to #1210 which is the same issue for Kafka), but the general point applies to Kinesis as well.

(FYI the Kafka version of this is a 0.3 priority due to a prospect ask, but Kinesis prioritization should follow Kinesis prospect requests.)

@frankmcsherry opened a general musing ticket for this: https://github.com/MaterializeInc/materialize/issues/2862

Just to clarify: #2862 is about the ability to fast-forward the updates in a stream, rather than starting much of the way through a stream. They might be related, but they could also be very different. The default implementation of #2862 would be to read everything in a stream and just advance the timestamps found there, rather than skipping any of the stream.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

pikulmar picture pikulmar  路  4Comments

JLDLaughlin picture JLDLaughlin  路  6Comments

johnjmartin picture johnjmartin  路  3Comments

benesch picture benesch  路  7Comments

JLDLaughlin picture JLDLaughlin  路  7Comments