Azure-cosmos-dotnet-v3: Retries to 429 never succeed under certain configurations

[Blackbaud-JonathanBell]

Describe the bug
Under specific configuration, the Cosmos V3 client will fail to succeed on any number of retries to a 429 response.

To Reproduce
All of the following configurations must be true to reproduce:

• Use V3 Cosmos Client
• Use ConnectionMode of Direct / TCP (this is the default)
• Configure Cosmos account to use multiple regions
• Configure Cosmos account for strong consistency
• Configure a very generous amount of retries, with either Polly via a custom RequestHandler, or the built in retry configuration:
  cosmosClientBuilder.WithThrottlingRetryOptions(new TimeSpan(0, 5, 0), 400);

Next, produce enough load via UpsertItemAsync or ExecuteStoredProcedureAsync to receive 429 responses from Cosmos.

Expected behavior

• Expect that some requests will receive a 429 and attempt some number of retries before succeeding.
• Using the RequestHandler, adding metrics to the retries would show that fewer requests took 2 retries vs 1, and fewer took 3 vs 2, etc.

This expected behavior is observed if any of the above configurations are changed.

Actual behavior

• Any request that receives a 429 will never succeed on subsequent retry attempts, even over the course of many minutes and many retries.
• All requests that receive a 429 will iterate through all retry attempts, continually failing, until it is out of retry attempts and then the overall request ends with an exception.

Environment summary
Microsoft.Azure.Cosmos Version: 3.3.2

.NET Core SDK (reflecting any global.json):
Version: 2.1.504
Commit: 91e160c7f0

Runtime Environment:
OS Name: Windows
OS Version: 10.0.15063

Additional context

• I only reproduced this with either Upserts or calling Stored Procs. I did not get enough load with reads to trigger 429s.
• This works correctly in the v2 client, even when set to Direct / TCP.
• If I use retries or polly outside of the Cosmos client, then it has no problem. That is, I can wrap UpsertItemAsync with a retry, and those retries will behave fine. Its only a problem for retries within the client (via the built in policy or via RequestHandler).

[j82w]

What setting are you using for the v2 client?

[j82w]

Can you also provide the full exception you are getting back? I did a quick test and I did not get any exceptions with a 429 status code, but I did get several 408 request timeouts.

[Blackbaud-JonathanBell]

What setting are you using for the v2 client?

For V2 client, I am using:

            var client = new DocumentClient(new Uri($"https://{accountName}.documents.azure.com:443/"), accountKey, new ConnectionPolicy
            {
                RetryOptions = new RetryOptions
                {
                    MaxRetryAttemptsOnThrottledRequests = 400,
                    MaxRetryWaitTimeInSeconds = 300
                },
                ConnectionMode = ConnectionMode.Direct,
                ConnectionProtocol =  Protocol.Tcp
            });

[Blackbaud-JonathanBell]

Can you also provide the full exception you are getting back? I did a quick test and I did not get any exceptions with a 429 status code, but I did get several 408 request timeouts.

Here is the exception:

Unhandled Exception: System.AggregateException: One or more errors occurred. (Response status code does not indicate success: 429 Substatus: 3200 Reason: ({
"Errors": [
"Request rate is large"
]
}).) ---> Microsoft.Azure.Cosmos.CosmosException: Response status code does not indicate success: 429 Substatus: 3200 Reason: ({
"Errors": [
"Request rate is large"
]
}).
at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode()
at Microsoft.Azure.Cosmos.CosmosResponseFactory.ToObjectInternalT
at Microsoft.Azure.Cosmos.CosmosResponseFactory.b__12_0T
at Microsoft.Azure.Cosmos.CosmosResponseFactory.ProcessMessageAsyncT
at CosmosV3ClientThrottleTest.Program.UpsertProfileWithSproc(Container container, Profile profile) in C:\Users\Jonathanbe\source\repos\CosmosV3ClientThrottleTest\CosmosV3ClientThrottleTest\Program.cs:line 292

[j82w]

How many request are you doing? Is it the same request rate for both v2 and v3? V3 has a lot of optimizations and should be able to handle a higher request rate than the v2.

I did 4000 upserts of new documents and did not get any 429s on a 400 RU container. Please take a look at my test and let me know if you have any suggestions.

[Blackbaud-JonathanBell]

@j82w Is this test connecting to an emulator or a real cosmos instance?

If a real one, is it configured with multiple regions and strong consistency? If not, then I wouldn't expect this to throw the exception because it would successfully complete within the generous retries your allowing it to handle internally (up to 5 minutes, up to 400 retries for each upsert request).

[Blackbaud-JonathanBell]

@j82w

I've tweaked the code in your example and was able to reproduce it when I use a Cosmos instance that is multi-region, strong consistency.

https://gist.github.com/Blackbaud-JonathanBell/e54b184d7d7c9cccfd3cfe98f9696561

[j82w]

I tested against a real Cosmos DB instance. I verified the test did hit 429, and all succeeded on the retries. I did not try strong consistency or multiple regions.

[Blackbaud-JonathanBell]

I also do not reproduce this unless I have enabled both strong consistency and multiple regions. I think the issue only surfaces under those conditions.

[j82w]

Having a strong consistency and multiple regions it might be exceeding the 5 minute timeout at that point. Each write is going to take significantly longer because it has to verify that it was replicated to all the other regions.

[Blackbaud-JonathanBell]

Having a strong consistency and multiple regions it might be exceeding the 5 minute timeout at that point. Each write is going to take significantly longer because it has to verify that it was replicated to all the other regions.

I don't think the potential delays in a multi-region write means that there isn't a clear bug here:

• Multi-region, strong works if you use Gateway mode instead of Direct.
• Multi-region, strong works if you use Direct and the 2.0 client instead of 3.0 client.
• Multi-region, strong works with the 3.0 client --- the only problem is if you have a 429 response, then an unlimited number of retries within that client request will always fail. If you turn off all of the load, and then issue a new request during the minutes while the original request continues to retry and fail, the new request will go through fine. (I've tested this)

The issue is the nature of the retries internal to the client are always given the same original 429 outcome (in direct mode) even when there is no RU load in Cosmos, and new requests (retries wrapping the client vs internal to the client) will work fine b/c its an intrinsically new request rather than a tainted internal retry.

[Blackbaud-JonathanBell]

To illustrate this further, I've updated my gist with a different test. It uses Polly to retry 429 errors every 10 seconds, for up to 20 minutes.

The test attempts 50 upserts to Cosmos, and logs successes or failures for each retry -- including the retry count that it was one.

I ran the test and included the output in the gist as well (link below). Almost immediately, 47 requests completed successfully on the first try. The other 3 upserts hit a 429 and had to retry.

The test then slowly ran, retrying those 3 upserts every ~10 seconds over the course of 20 minutes, continually failing each time, and then completed without success for any of these 3 failed upserts.

https://gist.github.com/Blackbaud-JonathanBell/e54b184d7d7c9cccfd3cfe98f9696561#file-output-txt

[j82w]

Thanks for the repo, and clarification. I'm able to repo it using the code you provided, but I'm having issues getting it repo against the current master. It seems to be throwing 408 exception instead now. I'm still investigating.

[alexmartinezm]

Hi there, is there any update about this issue? I'm facing with the same bug. Thanks in advance.

[ylibrach]

We seem to be experiencing this same problem as well. We use Polly as a resilience provider, and we noticed that when we retry to move the iterator on the IAsyncEnumerable on 429 failures as part of a call to CosmosContainer.GetItemQueryIterator() (we use v4), the SDK gets stuck and seems to continually rethrow the 429 exception. No subsequent requests to Cosmos are ever actually attempted to be made by the SDK.

I created some unit tests to mock responses from the SDK and confirmed that the resilience chain is working correctly. In other words, I created an IAsyncEnumerable which sometimes throws an exception and had the mock SDK return that. The resilience chain correctly retried on those exceptions, and was able to keep the iterator moving forward on retries. So based on this, it appears that the issue is within the Cosmos SDK itself.