Azure-cosmos-dotnet-v3: ReadNextAsync throws CosmosException (InternalServerError) when operation is canceled

[maxvasil]

Describe the bug
During query execution, ReadNextAsync function throws CosmosException with InternalServerError code if cancellation token is canceled.

To Reproduce
The following code reproduces the issue:

``` c#
string userId = ...;
Container container = ...;

FeedIterator feedIterator = container.GetItemQueryIterator(
new QueryDefinition($"SELECT TOP 10 * FROM Orders WHERE Orders.userId = \"{userId}\""),
requestOptions: new QueryRequestOptions { PartitionKey = new PartitionKey(userId) });

CancellationTokenSource cts = new CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromMilliseconds(10));

// Throws Microsoft.Azure.Cosmos.CosmosException with StatusCode = InternalServerError
await eventIterator.ReadNextAsync(cts.Token).ConfigureAwait(false);

**Expected behavior**
`System.OperationCanceledException` should be thrown.

**Actual behavior**
`Microsoft.Azure.Cosmos.CosmosException` with `StatusCode = InternalServerError` is thrown.

**Environment summary**
SDK Version: 3.6.0
OS Version: Windows

<details>
<summary>Stack trace</summary>

``` text
Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 500 Substatus: 0 Reason: (System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.Collections.AsyncCollection`1.<TakeAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducer.<TryMoveNextPageAsync>d__69.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.<TryMoveNextPageImplementationAsync>d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.<ExecuteWithSplitProofingAsync>d__52.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.ItemProducers.ItemProducerTree.<TryMoveNextPageAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosCrossPartitionQueryExecutionContext.<TryInitializeAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.OrderBy.CosmosOrderByItemQueryExecutionContext.<TryInitializeAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.OrderBy.CosmosOrderByItemQueryExecutionContext.<TryCreateAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.PipelinedDocumentQueryExecutionContext.<TryCreateAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<TryCreateSpecializedDocumentQueryExecutionContextAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<TryCreateFromPartitionedQuerExecutionInfoAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.<TryCreateCoreContextAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.<GetValueAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.LazyCosmosQueryExecutionContext.<ExecuteNextAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextWithNameCacheStaleRetry.<ExecuteNextAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CatchAllCosmosQueryExecutionContext.<ExecuteNextAsync>d__6.MoveNext()).
StatusCode = InternalServerError;
SubStatusCode = 0;
ActivityId = 00000000-0000-0000-0000-000000000000;
RequestCharge = 0;
{"Summary":{"StartUtc":"2020-03-16T17:18:18.2073179Z","UserAgent":"cosmos-netstandard-sdk/3.6.0|3.4.2|90261|X64|Microsoft Windows 10.0.17763 |.NET Framework 4.7.3610.0|"},"Context":[{"Id":"QueryReadNextAsync","ElapsedTime":"00:00:00.0415286"}]}

[ealsur]

You are creating a CancellationToken of 10ms. Queries do not have a 10ms SLA.

[ealsur]

@j82w should the OperationCanceledException be bubbled up? Looks like the AsyncCollection.TakeAsync is wrapping it on a HTTP 500.

[j82w]

This is a bug. Query is catching all exceptions and converting them to CosmosExceptions. It should just throw the OperationCanceledException instead.

[ealsur]

@bchong95 Should the code linked by Jake be letting the OperationCanceledException through?

[bchong95]

I think it should. But's up to you all how "exceptionless" you want that method to be.

[ealsur]

@j82w, @kirankumarkolli, should we let OperationCanceled bubble instead of converting it on a Response with CosmosException in these cases?

[j82w]

I think throwing it is the best option right now. In v3 I don't see a way to make it exceptionless without a breaking change or a confusing user experience.

1. CosmosException and response message both have a status code. If we default it to 500 for client exceptions like this I think it will cause confusion.
2. I believe every other stream endpoint beside query is just throwing it.

We do have a backlog item to catch this and convert it to some cosmos client exception so it keeps the diagnostics.

[FabianMeiswinkel]

My preference would be an OperationCanceledException with an inner exception of type CosmosException to preserve the diagnostics if any partial results are available.

[j82w]

What about internal CosmosOperationCanceledException that extends OperationCanceledException. That way we can override the ToString() logic to include the diagnostics always.