Azure-cosmos-dotnet-v3: Change Feed iterator for partition key value

[ealsur]

There might be a need to do Change Feed read iterations for a specific Partition Key Value, for example, in multi-tenant solutions where it would be interesting to get the Change Feed just for a particular PK Value.

[Zenuka]

I would really love this feature since we sometimes have a peek load for a specific client and we don't want it to affect other clients. We wanted to use cosmos db, change feed and the azure function binding for this but azure functions only scales per partition key range and we don't have that much load to justify having something like 50.000 RU's. Even then, 2 clients could reside on the same partition. Having an azure function instance with a lease on a partition key (instead of a key range) would solve this. For now we went with table storage and service bus :-(

[bartelink]

(this should probably be a separate issue; if the answer is anything other than "nope, not feasible", I can write up a separate Issue) is there any way in which one might expose a way to do server-side filtering of stuff - e.g. if there was a container containing a mix of data and metadata docs and you only want to process the data ones ?

[ealsur]

We are aware of the need to parallelize work beyond the Partition Key Range, and we are working towards finding a performant solution. While client-side filtering is possible, it would still imply a processing overhead that would be not desirable.

@bartelink There is no server-side filtering that can be applied, which is part of the original problem, so that is why any filtering would need to happen client-side.

[Zenuka]

Thanks for the quick response. We’re actually considering implementing this our selfs. Would that seem a lot of work to you? And would you be available for some assistance and an PR?

[ealsur]

We always review PRs, but the problem lies in that there is no amount of client-side code that will make the filtering happen on the server-side for RU consumption to be efficient. The goal that is being worked on is support server-side filtering to allow a more granular client-side querying, which would lead to the ability to generate leases at a smaller granularity. That would enable small containers to have multiple hosts running in parallel even though they might have a single Partition Key Range.

[Zenuka]

I see. I thought because this feature request (https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/20493154-per-partition-key-change-feed) was done, the server would be ready for it. Thanks for the explanation!

[ealsur]

That is a different scenario. The V2 SDK had an API that would let you do one query to the Change Feed and pass the PartitionKey value, but that is one query.

The Change Feed Processor acts by generating multiple listeners that work in parallel, each listener represents a Partition Key Range. There are backend APIs we can call to know how many Partition Key Ranges exist, and hence, create as many leases/listeners. But that is not possible with the Partition Key values, there is no backend API we can call to get all possible Partition Key values at this point.

[ealsur]

Closing this as its included in the new Change Feed pull model