Azure-cosmos-dotnet-v3: MalformedContinuationTokenException starting with SDK v3.7
We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.
Describe the bug
We are seeing consistent MalformedContinuationTokenException exceptions starting with SDK v3.7 that did not occur with the same query in SDK v3.6. The same issue happens with SDK v3.8 as well.
To Reproduce
This is a cross-partition query over Gateway mode and as we continue to drain more results from the FeedIterator the exception will be thrown on a call to await feed.ReadNextAsync(). It looks like it may work for the first set of results from every physical partition then throws as it goes back to get a second set of results from a physical partition (my guess). Looking at Fiddler there is no error coming back over the wire, it appears to be purely inside the SDK.
Expected behavior
The query to continue to return results as it did with SDK v3.6
Actual behavior
A MalformedContinuationTokenException is thrown with a reason "CompositeContinuationToken is missing field".
Environment summary
SDK Version: v3.7 and v3.8
OS Version: Windows
Additional context
Activity ID examples:
b0e3b405-f7e3-4c22-bf11-0a86fcd2782c
eed47b7d-f177-475b-849b-d0a5a67f4e77
Error message:
Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: ; Reason: (CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}});););););););); CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}}
Call stack:
at Microsoft.Azure.Cosmos.Query.Core.Monads.TryCatch`1.Try[T](Func`2 onSuccess)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.TryCreateAsync(ExecutionEnvironment executionEnvironment, CosmosQueryContext queryContext, CrossPartitionInitParams initParams, CosmosElement requestContinuationToken, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.TryCreateAsync(ExecutionEnvironment executionEnvironment, CosmosQueryContext queryContext, CrossPartitionInitParams initParams, CosmosElement requestContinuationToken, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateSpecializedDocumentQueryExecutionContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, List`1 targetRanges, String collectionRid, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateSpecializedDocumentQueryExecutionContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, List`1 targetRanges, String collectionRid, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateFromPartitionedQuerExecutionInfoAsync(PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, ContainerQueryProperties containerQueryProperties, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateFromPartitionedQuerExecutionInfoAsync(PartitionedQueryExecutionInfo partitionedQueryExecutionInfo, ContainerQueryProperties containerQueryProperties, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<>c__DisplayClass2_0.<Create>b__1(CancellationToken innerCancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Cosmos.FeedIteratorInlineCore`1.<>c__DisplayClass5_0.<ReadNextAsync>b__0()
at Microsoft.Azure.Cosmos.TaskHelper.RunInlineIfNeededAsync[TResult](Func`1 task)
at Microsoft.Azure.Cosmos.FeedIteratorInlineCore`1.ReadNextAsync(CancellationToken cancellationToken)
at ViralLaunch.Core.Services.AzureCosmosDBService.PerformFeedRetrieve[TOut](Func`3 feedBuilder, String queryText, Nullable`1 maxItemCount, String continuationTokenIn, Boolean trackMetrics)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.ExecuteNextAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.ExecuteNextAsync(CancellationToken token)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionComponent.SkipTake.TakeDocumentQueryExecutionComponent.ClientTakeDocumentQueryExecutionComponent.DrainAsync(Int32 maxElements, CancellationToken token)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.Parallel.CosmosParallelItemQueryExecutionContext.DrainAsync(Int32 maxElements, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.TryMoveNextPageAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.ExecuteWithSplitProofingAsync(Func`2 function, Boolean functionNeedsBeReexecuted, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
at System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
codecadwallader
All 20 comments
Can you provide a reproduce for the error. If not we can start with a sample for how you are draining your query and passing the continuation token.
As for the error message :
CompositeContinuationToken is missing field: 'token': {"range":{"min":"05C189CD6732","max":"05C18F5D153C"}
````
It looks like your continuation token is missing a field. It should be something like:
{
"token": "asdfsadfasdf",
"range":{"min":"05C189CD6732","max":"05C18F5D153C"
}
````
bchong95
on 10 Apr 2020
Thanks for the quick response. I took a look and the continuation token coming back from FeedIterator<TOut>.ReadNextAsync() appears to be the same in v3.6 as it is in v3.7:
v3.6
{
"top": 978,
"sourceToken": [
{
"range": {
"min": "05C1653DB928",
"max": "05C1753DB928"
}
}
]
}
v3.7
{
"top": 978,
"sourceToken": [
{
"range": {
"min": "05C1653DB928",
"max": "05C1753DB928"
}
}
]
}
The query we are trying to run is a SELECT TOP 1000 ... with a MaxItemCount of 100. The idea is we want to return the first 100 results, and allow for paging up to the next 900 results but not allow the user to keep going indefinitely.
Because we want to get back exactly 100 results (no more, no less) to support paging the way we drain results is something we worked out together with you last summer if you may recall over https://github.com/Azure/azure-cosmos-dotnet-v2/issues/692 when the SDK changed partition behavior from v2.2.2 to v2.2.3
Pseudo-real-code:
string continuationToken = null;
var results = new List<TOut>();
while (true)
{
var feed = Container.GetItemQueryIterator<TOut>(continuationToken, maxItemCount: 100 - results.Count);
while (feed.HasMoreResults)
{
var response = await feed.ReadNextAsync();
if (results.Count + response.Count > 100)
{
break; // Page has too many results, re-issue query with adjusted max item count
}
results.AddRange(response);
continuationToken = feed.HasMoreResults ? response.ContinuationToken : null;
}
if (continuationToken == null || results.Count == 100)
{
break;
}
}
Thanks for your help!
codecadwallader
on 13 Apr 2020
@codecadwallader quick question, is this happening when working with the Emulator (if so, which version?) or the live service?
ealsur
on 13 Apr 2020
@ealsur The live service
codecadwallader
on 13 Apr 2020
I wasn't able to reproduce this error. Can I get a console app that reproduces this issue? Thanks.
bchong95
on 13 Apr 2020
Sure, I've been working on that. I'm also not able to reproduce it building up from a vanilla app so I'll need to spend some more time paring down our app and see what setting/aspect is making the difference.
codecadwallader
on 14 Apr 2020
Ok, I've isolated the issue down to Newtonsoft.Json.NullValueHandling.Ignore. We've had this property set with our JSON serialization for years since it helps reduce the payload sizes stored in Cosmos. The token property is indeed coming back as null and being dropped out of the ContinuationToken. I suspect this behavior has always been present, but perhaps SDK v3.7 is no longer handling the absence of token when sending it back in the same way it was in SDK v3.6?
If you would still like a minimal reproducible solution I can provide it.. but it's probably easier for you to just feed this one line into your own test bed:
JsonConvert.DefaultSettings = () => new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore };
Thanks for your help!
Addendum: This does not happen with every attempted query. It requires a query that will attempt to utilize the continuation token to get more results.
codecadwallader
on 15 Apr 2020
@bchong95 Just sanity checking to see if you were notified of my second update re: NullValueHandling. If you'd still like the minimal reproducible solution I can provide it.
codecadwallader
on 29 Apr 2020
@bchong95 any update on this?
vsadams
on 10 Jul 2020
@bchong95. @j82w I have found the cause of this issue and a hacky workaround.
So the reason this is happening is due to the code in FeedRangeCompositeContinuation.
The issue is the classes use of JsonConvert from the Newtonsoft library. JsonConvert will use the default func handle for creating a serializer. So for instance if the consumer of the cosmos libs does something like...
JsonConvert.DefaultSettings = () =>
{
var settings = new JsonSerializerSettings();
settings.NullValueHandling = NullValueHandling.Ignore;
return settings;
};
Then the code in that class is going to attempt to deserialize\serialize the CompositeContinuationToken using ignore for the null value handling. The result is you will get a continuation token string without a token. then CompositeContinuationToken.TryCreateFromCosmosElement is going to end up throwing because the string no longer has a "token" value for any continuation token that returned a null token.
If you have code that is trying to drain the feed then fetching more, i.e draining the feed in a while loop then using the continuation token to get more results it will end up potentially throwing out.
I would like to make a suggestion that we utilize a static serializer in the FeedRangeCompositeContinuation class that ensures nulls are not ignored. EIther that, or making the TryCreateFromCosmosElement method skip the valiation checks for the properties.
I am more than happy to make a PR if you guys think thats best or I can leave it to you guys. I know as a whole, Newtonsoft is being pulled out of the libs so maybe just using the System.Text.Json serializer is best.
Here is my hacky workaround for anyone else that encounters this and needs a patch for the time being...
public static void SetDefaults()
{
JsonConvert.DefaultSettings = () =>
{
//apply internal defaults
var settings = SetDefaults(new JsonSerializerSettings());
//temp fix for cosmos
settings.Converters.Add(new CosmosConverter());
return settings;
};
}
public class CosmosConverter : JsonConverter
{
private const string CompositeContinuationTokenClass = "Microsoft.Azure.Cosmos.Query.Core.ContinuationTokens.CompositeContinuationToken";
public CosmosConverter()
{
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
//utilize the newtonsoft default serializer settings
var tempSerializer = new JsonSerializer();
tempSerializer.Serialize(writer, value);
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
=> throw new NotImplementedException();
public override bool CanRead => false;
public override bool CanConvert(Type objectType) => objectType?.FullName == CompositeContinuationTokenClass;
}
Thanks @codecadwallader for the tip.
vsadams
on 11 Jul 2020
@vsadams This is great feedback, but query does not use FeedRangeCompositeContinuation. Query seems to go through https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Query/v3Query/QueryIterator.cs#L80
which is the place where the Malformed error message is generated.
ealsur
on 22 Jul 2020
Just to be sure, I also checked FeedRangeCompositeContinuation serialization/deserialization with your suggested global settings, with this test:
JsonConvert.DefaultSettings = () =>
{
JsonSerializerSettings settings = new JsonSerializerSettings();
settings.NullValueHandling = NullValueHandling.Ignore;
return settings;
};
const string containerRid = "containerRid";
List<Documents.Routing.Range<string>> keyRanges = new List<Documents.Routing.Range<string>>()
{
new Documents.Routing.Range<string>("A", "B", true, false),
new Documents.Routing.Range<string>("D", "E", true, false),
};
FeedRangeInternal feedRangeInternal = new FeedRangeEPK(new Documents.Routing.Range<string>("A", "E", true, false));
FeedRangeCompositeContinuation token = new FeedRangeCompositeContinuation(containerRid, feedRangeInternal, keyRanges);
// {"V":0,"Rid":"containerRid","Continuation":[{"range":{"min":"A","max":"B"}},{"range":{"min":"D","max":"E"}}],"Range":{"min":"A","max":"E"}}
string serialized = token.ToString();
Assert.IsTrue(FeedRangeContinuation.TryParse(serialized, out FeedRangeContinuation feedRangeContinuation));
FeedRangeCompositeContinuation feedRangeCompositeContinuation = feedRangeContinuation as FeedRangeCompositeContinuation;
The serialization indeed does not have the token attribute within the Continuation attribute, but the deserialization works as expected and the resulting Token is null as expected.
FeedRangeCompositeContinuation does have a custom converter https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/Serializer/FeedRange/FeedRangeCompositeContinuationConverter.cs
The issue seems to be with the mechanism used on queries to serializer/deserialize its continuation (which is not FeedRangeCompositeContinuation).
ealsur
on 22 Jul 2020
@ealsur it is definitely possible that I am leading this astray. I have not pulled the source and debugged against it. I am purely going from reading the code in GitHub.
However, the class FeedRangeCompositeContinuation does have a converter but CompositeContinuationToken does not. So when FeedRangeCompositeContinuation does something like https://github.com/Azure/azure-cosmos-dotnet-v3/blob/725931ccb581ac1d3a6b2b4737b7296f42a19a35/Microsoft.Azure.Cosmos/src/FeedRange/Continuations/FeedRangeCompositeContinuation.cs#L271 where is serializes the CompositeContinuationToken into a string the property will be missing.
Again, very possible, I am wrong on this. I can pull code to double check but that was my inclination on why my work around worked.
vsadams
on 22 Jul 2020
We agree on the behavior of FeedRangeCompositeContinuation, the string that gets output does not have the "token" property if you alter the global Json settings. But what I'm saying is that it doesn't affect when FeedRangeCompositveContinuation calls TryParse, it works as expected (the Token property is null) so no errors happen there.
Now, going back to the original issue (the Malformed message), that is not related to FeedRangeCompositeContinuation, as queries do not use this class nor it is involved in the path that returns this error message.
ealsur
on 22 Jul 2020
Gotcha, that makes sense!
vsadams
on 22 Jul 2020
@ealsur I pulled the cosmos code and debugged it.
This is what I am seeing. This line here https://github.com/Azure/azure-cosmos-dotnet-v3/blob/f72a33dafcaec202c56da216172d99f2b98c7983/Microsoft.Azure.Cosmos/src/Query/Core/ExecutionContext/Parallel/CosmosParallelItemQueryExecutionContext.ContinuationToken.cs#L44 is attempting to use a default serializer settings. However, due to the func handle attached to JsonConvert.DefaultSettings, the default settings are overriden.
In order to do a simple test to make sure this theory is correct, I removed my work around in my code base and tested it. I did get a malformed token originating there and eventually got the exception. I then modified the code and used a static serializer like
private static JsonSerializer Serializer = new JsonSerializer();
protected override string ContinuationToken
{
get
{
IEnumerable<ItemProducer> activeItemProducers = this.GetActiveItemProducers();
string continuationToken;
if (activeItemProducers.Any())
{
IEnumerable<CompositeContinuationToken> compositeContinuationTokens = activeItemProducers.Select((documentProducer) => new CompositeContinuationToken
{
Token = documentProducer.CurrentContinuationToken,
Range = documentProducer.PartitionKeyRange.ToRange()
});
StringBuilder sb = new StringBuilder();
using StringWriter sringWriter = new StringWriter(sb);
using JsonWriter jsonTextWriter = new JsonTextWriter(sringWriter);
Serializer.Serialize(jsonTextWriter, compositeContinuationTokens);
continuationToken = sb.ToString();
//continuationToken = JsonConvert.SerializeObject(compositeContinuationTokens);
}
else
{
continuationToken = null;
}
return continuationToken;
}
}
(Not pretty it was just a test)..but this worked.
Hope this is helpful!
vsadams
on 23 Jul 2020
@bchong95 Can you verify what @vsadams is mentioning? Is this a possible fix?
ealsur
on 23 Jul 2020
@ealsur any update on whether or not this is getting in?
vsadams
on 7 Oct 2020
@bchong95 can you help verifying the scenario @vsadams is providing? Would his fix work? If so, I'm guessing you can send a PR @vsadams ?
ealsur
on 7 Oct 2020
@ealsur or @bchong95 any update?
vsadams
on 11 Nov 2020
Related issues
Liversage
路
7Comments
matthewhancock
路
8Comments
miiitch
路
5Comments
Jubast
路
6Comments
divega
路
5Comments
Most helpful comment
@ealsur I pulled the cosmos code and debugged it.
This is what I am seeing. This line here https://github.com/Azure/azure-cosmos-dotnet-v3/blob/f72a33dafcaec202c56da216172d99f2b98c7983/Microsoft.Azure.Cosmos/src/Query/Core/ExecutionContext/Parallel/CosmosParallelItemQueryExecutionContext.ContinuationToken.cs#L44 is attempting to use a default serializer settings. However, due to the func handle attached to JsonConvert.DefaultSettings, the default settings are overriden.
In order to do a simple test to make sure this theory is correct, I removed my work around in my code base and tested it. I did get a malformed token originating there and eventually got the exception. I then modified the code and used a static serializer like
(Not pretty it was just a test)..but this worked.
Hope this is helpful!