Azure-cosmos-dotnet-v3: FeedIterator<T>.ReadNextAsync throws NullReferenceException with custom serializer

[fuocor]

Describe the bug
FeedIterator.ReadNextAsync throws NullReferenceException with custom serializer.
The stream sent to the serializer is empty. `All the other Cosmos methos serialize fine and GetItemQueryStreamIterator works as does the default serializer.

To Reproduce

var families = new List<Family>();

var setIterator = container.GetItemQueryIterator<Family>(requestOptions: new QueryRequestOptions { MaxItemCount = 1 });
while (setIterator.HasMoreResults)
{
    foreach (var item in await setIterator.ReadNextAsync())
    {
        families.Add(item);
    }
}

Expected behavior
Deserialized object.

Actual behavior
NullReferenceException

Environment summary
SDK Version: 3.0.100
Cosmos Version: 3.2.0
OS Version: Windows 10

[j82w]

Can you please provide the full exception message?

Query does return empty pages. If you have a custom serializer it should be able to handle this.

[fuocor]

I tracked it down, The FeedIterator wraps the result inside a response with a "Documents" element. The default serializer seems to address it whereas a custom serializer needs to drill into the response

[j82w]

Here is what the contract should look like.

            // The stream contract should return the same contract as read feed.
            // {
            //    "_rid": "qHVdAImeKAQ=",
            //    "Documents": [{
            //        "id": "03230",
            //        "_rid": "qHVdAImeKAQBAAAAAAAAAA==",
            //        "_self": "dbs\/qHVdAA==\/colls\/qHVdAImeKAQ=\/docs\/qHVdAImeKAQBAAAAAAAAAA==\/",
            //        "_etag": "\"410000b0-0000-0000-0000-597916b00000\"",
            //        "_attachments": "attachments\/",
            //        "_ts": 1501107886
            //    }],
            //    "_count": 1
            // }

[j82w]

@fuocor I'm closing this issue since you figured out the issue. Please re-open it the issue isn't fixed.

[fuocor]

It makes writing a custom serializer nearly impossible. The type requested from the serializer is CosmosFeedResponseUtil<T>. How do you expect a custom serializer to instantiate that?

[j82w]

That's a good point. The test have access to the internals which is why it's not having an issue with this.

The only option I see is to make CosmosFeedResponseUtil<T> public. Do you see any other alternatives?

[fuocor]

Ask for an interface or abstract base class.

[fuocor]

You have similar problems with ChangeFeed Processing.

 ResponseMessage.EnsureSuccessStatusCode()
    CosmosResponseFactory.ToObjectInternal[T](ResponseMessage cosmosResponseMessage, CosmosSerializer jsonSerializer)
    CosmosResponseFactory.<CreateItemResponseAsync>b__6_0[T](ResponseMessage cosmosResponseMessage)
    CosmosResponseFactory.ProcessMessageAsync[T](Task`1 cosmosResponseTask, Func`2 createResponse)
    DocumentServiceLeaseStoreCosmos.MarkInitializedAsync()
    BootstrapperCore.InitializeAsync()
    BootstrapperCore.InitializeAsync()
    PartitionManagerCore.StartAsync()
    ChangeFeedProcessorCore`1.StartAsync()

[j82w]

ResponseMessage and ItemResponse are public. It shouldn't have any issue creating an instance of these.

[fuocor]

It's the content of the response that is the problem DocumentServiceLeaseCore and LockDocument

[j82w]

DocumentServiceLeaseCore and LockDocument shouldn't be using the custom serializer. It should be using the property serializer. That's definitely a bug. Fixing that seems like it's going to be interesting.

[joshlang]

FYI: Using cosmos change feed is completely blocked for us until this is fixed. We can't deserialize CosmosFeedResponseUtil.

[j82w]

@ealsur does PR #944 fix this issue?

[ealsur]

Yes, it fixes using a custom serializer with Change Feed, as it will only used for the Delegate changes.

[j82w]

We should be doing a release later this week or early next week. Just waiting on a few more PRs to get merged first.

[joshlang]

That timeline will work for us. Thanks!