Delta: Read stream cursor on Delta table does not correctly set, remains at startingVersion: true

[AlexDHoffer]

Hi folks,

We have a read stream attached to a Delta table that will not recognize or accept a starting cursor. The stream is setup like so:

stream = ( spark.readStream .format('delta') .option('startingVersion', 'latest') .load('/mnt/table') )

The stream insists on starting from the first data and processing the whole table, but never makes it past the first batch, and keeps resetting the cursor. We have evidence of this as every batch that finishes and flushes a checkpoint offset shows:

"isStartingVersion":true

We have other streams attached to the same Delta table that work perfectly, and show "isStartingVersion":false.
We have tried clearing the checkpoint directory and tried using both startingVersion and startingTimestamp but to no avail - it always gets stuck on the same rows.

We are on Databricks Runtime 7.4.

[morrifeldman]

We are seeing what seems like the same issue while using open source delta built from the master branch. We set "startingVersion" to "latest" but the stream continually resets to the beginning. In the checkpoint files we also see that they all say "isStartingVersion": true. In our case the problem only occurs when setting a trigger limit, e.g. maxBytesPerTrigger or maxFilesPerTrigger.

I think this failing test demonstrates the bug:

https://github.com/morrifeldman/delta/commit/77cad15115593d7611b3ff75c3585e6d841a469e

Output from test is that 10 and 11 are duplicated, and 5-9 are read by mistake.

== Results ==
!== Correct Answer - 5 ==   == Spark Answer - 12 ==
!struct<>                   struct<id:bigint>
 [10]                       [10]
![11]                       [10]
![12]                       [11]
![13]                       [11]
![14]                       [12]
!                           [13]
!                           [14]
!                           [5]
!                           [6]
!                           [7]
!                           [8]
!                           [9]

[zsxwing]

Thanks a lot for reporting this and the great test. I found a bug related to startingVersion. Will fix it soon.