Azure-cosmos-dotnet-v3: Change Feed Processor: Add support for manual checkpoint

[ealsur]

Full detail of user request is on #616

TL;DR

The request is to add manual checkpoint as a configuration option to the Change Feed Processor, this was a feature already available in V2 CFP through a configuration option.
The lack of this feature is a migration blocker for users using it on CFP V2.

How will the user enable the configuration

We can extend the ChangeFeedProcessorBuilder to explicitly have an option WithManualCheckpoint:

ChangeFeedProcessor changeFeedProcessor = sourceContainer
            .GetChangeFeedProcessorBuilder<ToDoItem>(processorName: "myProcessor", HandleChangesAsync)
                .WithInstanceName("myInstanceName")
                .WithLeaseContainer(leaseContainer)
                .WithManualCheckpoint();
                .Build();

How will the user interact with the checkpoint mechanism

Similar to V2 CFP, the Handler will receive a Context object:

    public abstract class ChangeFeedProcessorContext
    {
        /// <summary>
        /// Gets the token representative of the current lease from which the changes come from.
        /// </summary>
        public abstract string LeaseToken { get; }

        /// <summary>
        /// Gets the session token returned as part of the Change Feed request.
        /// </summary>
        public abstract string SessionToken { get; }

        /// <summary>
        /// Checkpoints progress of a stream. This method is valid only if manual checkpoint was configured.
        /// Client may accept multiple change feed batches to process in parallel.
        /// Once first N document processing was finished the client can call checkpoint on the last completed batches in the row.
        /// In case of automatic checkpointing this is method throws.
        /// </summary>
        /// <exception cref="LeaseLostException">Thrown if other host acquired the lease or the lease was deleted.</exception>
        public abstract Task CheckpointAsync();
    }

• The LeaseToken will help with users who want to send monitoring/diagnosing information to understand which is the lease where the changes are coming from. It also helps to send telemetry that identifies which leases are being processed.
• SessionToken is useful if the user is sending the changes to another system that has a different client instance. For example, (real customer scenario) once the changes are received, they are sent to a Queue where they get picked up with another application and used the information to read the document. If the user is on Session consistency, without the Session Token, those reads might fail with a 404.
• CheckpointAsync is the API meant to be called after the user's custom logic has decided that it is time to checkpoint.

Can the checkpoint fail?

Yes, there is a possibility of the call to CheckpointAsync to fail in expected scenarios:

• The current lease has been deleted externally (this is not a common scenario but if the user manually deletes the documents, it can happen).
• The lease was acquired by another host. In a load-balancing scenario it can occur that the lease has been taken by another host.

How would the user call the CheckpointAsync?

public async Task HandleChangesAsync(
    ChangeFeedProcessorContext context,
    IReadOnlyCollection<ToDoItem> changes, 
    CancellationToken cancellationToken)
{
    Console.WriteLine($"Started handling changes for lease {context.LeaseToken}...");
    foreach (ToDoItem item in changes)
    {
        // put changes in some buffer using context.LeaseToken as buffer key
    }

    if (buffer[context.LeaseToken].Size == expectedMax)
    {
        try
        {
            await context.CheckpointAsync();
        }
        catch(ChangeFeed.LeaseLostException ex)
        {
            Console.WriteLine($"Checkpoint failed for {context.LeaseToken} due to the lease being transfered to another host.");
            throw; // propagate the exception to shutdown the local processing of the lease
        }
    }

    Console.WriteLine($"Finished handling changes for lease {context.LeaseToken}.");
}

Questions

• What happens if the user switches on ManualCheckpoint and uses a delegate without the context? Should we throw a runtime exception? Or would documentation be enough? (manual checkpoint being a feature that needs to be turned on manually and its documentation should cover that)

[joshidp]

Hi @ealsur ,

Can you please share the release timeline for manual check-point feature.

Thanks

[bartelink]

Sorry for pestering but.... bump ;)

Any signs of this getting prioritized in the medium term?

[bartelink]

Is there any roadmap and/or indicative information of any kind available?

The absence of this feature is a major blocker for adopting the V3 SDK, which is causing significant concern for multiple projects.

[ealsur]

@bartelink I am terribly sorry for the delay, priorities shifted in this past year and I still have to follow up and get approval of this API. We are tracking this, but the work to enable higher scale (you might have seen the PRs about it) took higher priority.

[bartelink]

Thanks @ealsur - definitely not trying to pin this on you; its abundantly clear you're a) snowed under b) but still doing great work !

My main concern is that this remain on the agenda - while its theoretically possible to rewrite our consumer loops in terms of the CFP Pull Model, that's a disaster scenario from the point of view of upgrading our systems with any degree of confidence. i.e. while we believe we're in a good place to upgrade from V2 to V3, having to simultaneously move from V2 to V3 and from the standard CFP model with explicit checkpointing to Pull Model would be a massive amount of change in one go.

_TL;DR Having feature parity between V2 and V3 (aka having the Explicit Checkpoint mechanism reinstated as per the V2 CFP) is not a nice to have for anyone that is already working with 'higher scale' V2 system which is reliant on being able to control that aspect to deliver it_

[ealsur]

I whole heartily agree, and I am keeping this one on my mind, and working on it every time I have some free time. It is being tracked in our agenda, so it won't be forgotten.