Azure-cosmos-dotnet-v3: Changing default CosmosSerializationOptions to camelCase causes an exception in CosmosDB SDK V4

[GSmanXVI]

Describe the bug
I have a problem with a CosmosDB SDK V4.0.0-preview3. When I try to change default CosmosSerializationOptions it causes an exception when I try to use GetItemQueryIterator. Everything else except this method works fine.

To Reproduce
``` C#
var client = new CosmosClientBuilder(connectionString)
.WithSerializerOptions(new CosmosSerializationOptions { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase })
.Build();
var container = client.GetContainer("SocialNetwork", "Posts");
var query = new QueryDefinition("SELECT * FROM c");
await foreach (var item in container.GetItemQueryIterator(query))
{
Console.WriteLine(item);
}

**Actual behavior**
Exception message:

Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'source')
at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
at System.Linq.Enumerable.ToListTSource+MoveNext()
at Azure.Cosmos.PageResponseEnumerator.FuncAsyncPageable1.AsPages(String continuationToken, Nullable1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext() at Azure.AsyncPageable1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at ConsoleAppCosmosDb.Program.Main(String[] args) in C:UsersXPSsourcereposConsoleAppCosmosDbConsoleAppCosmosDbProgram.cs:line 29
at ConsoleAppCosmosDb.Program.Main(String[] args) in C:UsersXPSsourcereposConsoleAppCosmosDbConsoleAppCosmosDbProgram.cs:line 29
at ConsoleAppCosmosDb.Program.

(String[] args)
```
If I remove this option or change CamelCase to Default everything works.

Environment summary
SDK Version: 4.0.0-preview3
OS Version: Windows 10

[j82w]

@ealsur is this a current limitation or is this a bug?

[ealsur]

@GSmanXVI are you using any System.Text.Json converters in your Post class? Or are you using any Newtonsoft.Json decorators still?

[ealsur]

Looks like this happens due to the fact that V4 still uses the CosmosFeedResponseUtil

[arkiaconsulting]

One can use GetItemQueryStreamIterator as a temporary workaround

[jongio]

With this code:

CosmosClientOptions options = new CosmosClientOptions
{
  SerializerOptions = new CosmosSerializationOptions { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase }
};

CosmosClient cosmosClient = new CosmosClient(
  Environment.GetEnvironmentVariable("AZURE_COSMOS_ENDPOINT"),
  Environment.GetEnvironmentVariable("AZURE_COSMOS_KEY"), options);

CosmosContainer cosmosContainer = cosmosClient.GetDatabase(Environment.GetEnvironmentVariable("AZURE_COSMOS_DB")).GetContainer(Environment.GetEnvironmentVariable("AZURE_COSMOS_CONTAINER"));
QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM c");
List<QueueMessage> msgs = new List<QueueMessage>();
await foreach (QueueMessage item in cosmosContainer.GetItemQueryIterator<QueueMessage>(queryDefinition))
{
  msgs.Add(item);
}

Without options all props are null:

With options:

CosmosClientOptions options = new CosmosClientOptions
{
  SerializerOptions = new CosmosSerializationOptions { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase }
};

Get this exception

Exception has occurred: CLR/System.ArgumentNullException
An exception of type 'System.ArgumentNullException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'Value cannot be null.'
   at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponseWithSerializer[T](Response cosmosResponseMessage, CosmosSerializer serializer)
   at Azure.Cosmos.CosmosResponseFactory.CreateQueryFeedResponse[T](Response cosmosResponseMessage)
   at Azure.Cosmos.PageIteratorCore`1.<GetPageAsync>d__3.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Azure.Cosmos.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.Cosmos.PageResponseEnumerator.FuncAsyncPageable`1.<AsPages>d__2.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.GetResult(Int16 token)
   at Azure.AsyncPageable`1.<GetAsyncEnumerator>d__6.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at cosmostest.Program.<Main>d__0.MoveNext() in C:\temp\cosmostest\Program.cs:line 34
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at cosmostest.Program.<Main>d__0.MoveNext() in C:\temp\cosmostest\Program.cs:line 34

Workaround 1 - Don't use SerializerOptions, use JsonPropertyName attributes

public class QueueMessage
{
    [JsonPropertyName("id")]
    public string Id { get; set; }
    [JsonPropertyName("uid")]
    public string Uid { get; set; }
    [JsonPropertyName("imageUri")]
    public string BlobUri { get; set; }
    public string ImageText { get; set; }
    [JsonPropertyName("sentiment")]
    public string Sentiment { get; set; }
}

Workaround 2 - Use GetItemQueryStreamIterator

GetItemQueryStreamIterator returns json like this

{
   "Documents": []
}

So you need to create a new class with a Documents property. See QueryStream class below.

You also need to use CosmosPropertyNamingPolicy.Default in options for the Documents property.

But you need to use new JsonSerializerOptions() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase } when you read the stream.

using System;
using System.Collections.Generic;
using System.IO;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading.Tasks;
using Azure;
using Azure.Cosmos;
using Azure.Cosmos.Serialization;
using DotNetEnv;
namespace cosmostest
{
    public class QueryStream
    {
        [JsonPropertyName("Documents")]
        public QueueMessage[] Documents { get; set; }
    }

    class Program
    {
        static async Task Main(string[] args)
        {
            Env.Load();

            CosmosClientOptions options = new CosmosClientOptions
            {
                SerializerOptions = new CosmosSerializationOptions { PropertyNamingPolicy = CosmosPropertyNamingPolicy.Default }
            };

            CosmosClient cosmosClient = new CosmosClient(
                        Environment.GetEnvironmentVariable("AZURE_COSMOS_ENDPOINT"),
                        Environment.GetEnvironmentVariable("AZURE_COSMOS_KEY"), options);

            CosmosContainer cosmosContainer = cosmosClient.GetDatabase(Environment.GetEnvironmentVariable("AZURE_COSMOS_DB")).GetContainer(Environment.GetEnvironmentVariable("AZURE_COSMOS_CONTAINER"));
            QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM c");
            List<QueueMessage> msgs = new List<QueueMessage>();

            await foreach (Response response in cosmosContainer.GetItemQueryStreamIterator(queryDefinition))
            {

                var queryStream = await JsonSerializer.DeserializeAsync<QueryStream>(response.ContentStream,
                    new JsonSerializerOptions() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });

                msgs.AddRange(queryStream.Documents);
            }

        }
    }
}

azsdke2e

[eskaufel]

So I ran into this bug today, and I got somewhat surprised. This bug is getting close to 1 year old, not patched, and a part of the vNext release.

What is going on? When is the vNext SDK to be released? Has your focus changed to some other product?

[ealsur]

@eskaufel I understand the concern, currently V4 is an experimental version, not meant for production usage, and the efforts are mainly put to harden and improve V3, which acts as a base of V4. There is no ETA for V4 GA.

Since V4 is only experimental and preview, why are you currently using it? Are you doing some POC? Why not use V3 SDK instead? What are the reasons you are currently using V4?

[eskaufel]

I was playing with an idea. I often use preview when I know the timeline is considerably into the future. I was also hoping to use just one serializer across the project.
In the end, the name preview3 made me think v4 was closer to a release than it is.