Polly: HttpClient: timeout-per-try versus timeout across whole operation

[abjrcode]

Summary:

I have a typed http client and I have changed the default timeout on that client.
I configured a policy to handle transient errors and OperationCanceledExceptions
and retry.
I am still getting a TaskCanceledException though

Expected behavior:
The TaskCanceledException is caught and handled as per the configured policy
Actual behaviour:
Exception bubbles up and causes http request to fail

Steps / Code to reproduce the problem:
In my Startup class:

var githubHttpClientBuilder = services
                .AddHttpClient<GitHubClient>()
                .AddTransientHttpErrorPolicy(builder =>
                    builder
                        .Or<TaskCanceledException>()
                        .OrResult(msg => msg.StatusCode == HttpStatusCode.ServiceUnavailable)
                        .OrResult(msg => msg.StatusCode == HttpStatusCode.GatewayTimeout)
                        .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))));

Now inside my GitHubClient constructor I set the following on the received HttpClient:
httpClient.Timeout = TimeSpan.FromSeconds(3);

Now in some WebApi controller that I have where the GitHubClient is injected into the constructor, I do the following:

_githubClient.PostAsync...

Then I found this and implemented this based on it:

TimeoutPolicy timeout = Policy.TimeoutAsync(3);
            RetryPolicy<HttpResponseMessage> retry = HttpPolicyExtensions
                .HandleTransientHttpError()
                .Or<TimeoutRejectedException>()
                .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

            // Wrap them in the other order if you want the timeout per call, not per overall strategy
            PolicyWrap<HttpResponseMessage> retryWithTimeout = timeout.WrapAsync(retry);

var teamcityHttpClientBuilder = services
                .AddHttpClient<TeamCityClient>()
                .AddPolicyHandler(retryWithTimeout);

and now I am getting TimeoutRejectedExceptions !

I expect that the previous snippets would behave the following:

1. Request is sent and a timeout happens if it takes more than 3 seconds
2. Polly captures that task canceled exception caused by the timeout and retries after 2^i where i = 3 now
3. Repeat the previous step if another 3 second time out occurs with i = i-1
4. Either succeed after all attempts or fail and return last HttpResponseMessage

[reisenberger]

Hi @abjrcode . Please see our documentation on HttpClientFactory and timeouts which covers this. Summary:

• httpClient.Timeout acts as an overall timeout across the whole execution (time for all tries and waits in between them). When that timeout happens, it throws OperationCanceledException. After this overall timeout httpClient.Timeout has occurred, no further retries of that execution can take place (the timeout has already cancelled the whole operation).
• Polly TimeoutPolicy can be used as a timeout-per-try. To do this, it must be configured _inside_ the RetryPolicy in the PolicyWrap or sequence of HttpDelegatingHandlers. When TimeoutPolicy triggers a timeout, it throws TimeoutRejectedException.

---

So to obtain the expected behaviour stated in your question, keep your definitions of timeout and retry the same, but then either (this example uses PolicyWrap):

var teamcityHttpClientBuilder = services
            .AddHttpClient<TeamCityClient>()
            .AddPolicyHandler(retry.WrapAsync(timeout)); // Note: retry.WrapAsync(timeout), ie the opposite order from in your question.

Or equivalently (this example uses nested DelegatingHandlers):

var teamcityHttpClientBuilder = services
            .AddHttpClient<TeamCityClient>()
            .AddPolicyHandler(retry)
            .AddPolicyHandler(timeout);

In addition, you can:

teamcityHttpClient.Timeout = /* whatever timeout you want for the whole operation, the combined time allowed for all tries and waits between tries */ ;

---

The behaviour will be as follows, compared to your last stated expectation 1 / 2 / 3 / 4:

1. Request is sent and a timeout happens if it takes more than 3 seconds

Yes.

1. Polly captures that ~task canceled exception~ TimeoutRejectedException caused by the timeout and retries after 2^i where i = 3 now

Yes but it starts with i = 1. And it will be a TimeoutRejectedException (this is what Polly's TimeoutPolicy throws), rather than TaskCanceledException. Polly intentionally throws a different exception, TimeoutRejectedException, so that timeout cancellation can be distinguished from other cancellation.

1. Repeat the previous step if another 3 second time out occurs with i = i-1

retryAttempt increases, so this will be with i = i + 1

4.Either succeed after all attempts or fail and return last HttpResponseMessage

Yes. Or it may fail and rethrow any exception which the last try failed with.

[abjrcode]

Sorry missed that part of the documentation 😞
Thank you for you help clarifying this 👍

[reisenberger]

No problem, you're welcome