Polly Policy.Handle(condition)

[subatta]

Can Polly support handling conditions in addition to an exception?

[reisenberger]

Hi @subatta !

Polly can handle a condition on an exception: Policy.Handle<SqlException>(ex => ex.Number == 1205)

Polly can handle a condition on a result returned by an execution: Policy .HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.NotFound)

Are either of these what you were looking for?

If not, please do explain more about the scenario you are thinking of, and we will try to help further.

[subatta]

Thanks @reisenberger : I'm looking for a retry until a condition is satisfied on a request not using an exception generic. For example, the following Policy usage retries with a wait until the status is Ok.

.HandleResult(r => r.StatusCode == HttpStatusCode.Ok)
.WaitAndRetry(...)

[reisenberger]

Thanks @subatta for that clarification and example.

The second option I highlighted (handle a condition on a result returned by an execution) should allow you to do what you have expressed.

You have expressed your condition as a positive you want to achieve, a success condition (you want to retry until status is ok). In Polly, we express result conditions for policies to handle as the negatives - the faults you want to handle. Just flip the polarity of your condition and it'll fit right into the handle syntax.

c# Policy .HandleResult<HttpResponseMessage>(r => r.StatusCode != HttpStatusCode.Ok) .WaitAndRetry(...)

Put in words: you don't want the policy to 'handle' anything when the status is ok; you want it to handle (to retry) when they are _not_ ok.

Note that you can mix handling exceptions and result codes, to cover both kinds of fault, as in the example shown at that link.

One note of caution sounds in my mind about .HandleResult<HttpResponseMessage>(r => r.StatusCode != HttpStatusCode.Ok); it feels like possibly a blunt instrument, _if_ the context is a general web context. There can be status codes returned in a general web context where a retry, with an identical call at least, won't yield any greater success (301; 307; 308; 400; 401; 403; 404 etc etc). Of course, I don't know the nature of the system you are calling - if the return codes you expect from the called system are more circumscribed, this may not be relevant. Or this may not be relevant if HttpStatusCode.Ok was only an example.

[reisenberger]

Note that we don't have a policy that handles a condition that _isn't_ based on the return value from the delegate executed.

But if you want the handle expression to be on a bool - if you're looking to express a kind of do until(true) { ... } loop with WaitAndRetry(...) built in - then you can do this, by expressing the work you want to do as a delegate which returns a bool success status. eg:

```c#
public bool DoMyWork(...) {
// ... my work
// return true (success) or false (failed)
}

The policy to `WaitAndRetry(...)` until the condition is met:
```c#
var doUntilTrueWithWaitAndRetry = Policy
    .HandleResult<bool>(false) // retry if DoMyWork() method returns false
    .WaitAndRetry(/* configuration */);

Used as:
c# doUntilTrueWithWaitAndRetry.Execute(() => DoMyWork());

Whether to make a further try will be determined both by the bool condition and by the number of retries you have configured in the .WaitAndRetry(...). If you want purely the bool condition to determine retries, you'd use .WaitAndRetryForever(...) in the above.

[subatta]

Thanks a lot @reisenberger . The use of this pattern is for Integration tests right now. I didn't look, I assumed (erroneously so) that the Handle: T is an exception since that's what all samples show.

.HandleResult<bool>(false) is exactly what I was looking for.

I'm glad we picked Polly and kudos to you and all developers!

[reisenberger]

Thanks @subatta for the kind words. And: you're welcome!

[subatta]

@reisenberger : quick question. How do I evaluate async predicate like HandleResult(myAsyncFunc)? Doing a HandleResult(myAsyncFund.Result) is blocking the policy.Execute indefinitely (likely a deadlock underneath).

[reisenberger]

@subatta First, Polly has async policies and overloads for asynchronous executions. You should be using these (if not already).

Second, we would not normally expect to see a complex evaluation (eg an awaited function evaluation) within .HandleResult(...). The intended pattern is to put constants or simple synchronous expressions there, eg .HandleResult(false). Complex functions are then executed through the relevant execute overload. For example, with async:

var doUntilTrueWithWaitAndRetry = Policy
    .HandleResult<bool>(false) // retry if delegate executed asynchronously returns false
    .WaitAndRetryAsync(/* configuration */) // Note ... Async() on end of method name.

await doUntilTrueWithWaitAndRetry.ExecuteAsync(() => DoMyWorkAsync()); // Polly's ExecuteAsync(...) overload will await DoMyWorkAsync() internally.

// where elsewhere:
public async bool DoMyWorkAsync(...) { 
   // ... my work using await statements
   // return true (success) or false (failed)
}

---

In general, yes, blocking on async code risks deadlocks. So if you are coding the equivalent of:

await Policy
    .HandleResult<bool>(myPredictAsync().Result)) // mixing blocking on async code ...
    .WaitAndRetryAsync(/* configuration */) 
    .ExecuteAsync(() => DoMyWorkAsync()); // ... with true async code

then yes, that will likely block.

This could also be expressed by saying Polly doesn't _support_ async handle predicates. If you have a particular use case, for which it only makes sense / makes particular sense for the .HandleResult<TResult>(...) predicate to be a more complex function executed asynchronously, could you please describe the scenario in more detail?

[subatta]

Sure @reisenberger: In an integration test, for example, messages are put on a bus and test executes remote(HTTP) calls few times until a max timeout is reached to check on a result. One of the conditions of evaluating success is looking into the HttpResponseMessage content which is accessible with async methods.

That evaluation within something like
HandleResult<HttpResponseMessage>(r => r.Content.ReadAsAsync<MyType>().Result.Any())
is where things get into trouble.

[reisenberger]

Hi @subatta , thanks for elaborating; that makes sense.

Since Polly doesn't support asynchronous executions in the handle predicates, to do this with Polly, you would need to refactor to move the work HttpResponseMessage.Content.ReadAsAsync<MyType>() into the delegate executed through the .ExecuteAsync(...) call.

If we can help any further with this, please do let us know!

[subatta]

Definitely, an example would be fantastic @reisenberger.

[reisenberger]

@subatta Sure. Can you provide a bit more context?

• the current policy you have;
• the method (at least full signature) you are currently executing through that policy (is this effectively some Func<HttpResponseMessage>?)
• the signature of ReadAsAsync<MyType>()

Thanks!

[subatta]

Sure. Here it is:

var retryPolicy = Policy
         .HandleResult<HttpResponseMessage>(
    r => r == null || !r.IsSuccessStatusCode|| !r.Content.ReadAsAsync<IEnumerable<MyType>>().Result.Any())
         .WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
var response = retryPolicy.Execute(() => apiConnector.ExecuteAndReturnResponse("/api/endpoint"));

apiConnector.ExecuteAndReturnResponse is a wrapper around HttpClient.

The expectation is that the retry will happen based on WaitAndRetry configuration until max attempts are reached or until the condition specified in HandleResult() is met.

Perhaps the condition evaluation can be done via Execute() (or the async ver), that'll be what I'm looking for.

[reisenberger]

@subatta What is the full signature of .ExecuteAndReturnResponse(string)?, ie is it:

HttpResponseMessage ExecuteAndReturnResponse(string url) // looks like this, from code you quoted?

or:

async Task<HttpResponseMessage> ExecuteAndReturnResponse(string url)

?

If ExecuteAndReturnResponse(string url) is blocking internally on any async calls, eg with something like HttpClient.GetAsync(...).Result, then changes around .ReadAsAsync<IEnumerable<MyType>>().Result may not be enough. Mixing sync and async code, or blocking on async code, is frequently a recipe for deadlocks. async code generally needs to be async all the way.

(I have a solution to move .ReadAsAsync<IEnumerable<MyType>>() out of your .HandleResult(...) clause, but answer to the above q might affect the solution.)

[subatta]

You're right there's no async overload for the HttpResponseMessage ExecuteAndReturnResponse(string url). I'd like to know the solution that is feasible while being minimally invasive. By that I mean, changes on your side, my side, in that order. :-)

[reisenberger]

hi @subatta .

If you have control over the HttpResponseMessage ExecuteAndReturnResponse(string url) code, it would be really helpful to know at this point whether internally it is doing something like HttpClient.GetAsync(...).Result. Otherwise, there is a risk that I spend time proposing solutions that will not help your scenario. Any next recommendation depends very much on the answer to this question.

[subatta]

Yes, it's a GetAsync(url).Result.

[reisenberger]

Hi @subatta I'm now clear the codebase under discussion (a) has no async/await (so a Polly ExecuteAsync() solution from me would be inappropriate unless you wholesale refactor to async/await); (b) is synchronously blocking on async calls in two places; (c) is using the synchronous Polly policy .WaitAndRetry(...) with synchronous .Execute(...)

Knowing (a) and (c) means the deadlock issue is unrelated to the use of Polly (there is nothing we could change in Polly that would unblock these deadlocks). The aspects of Polly you are invoking in (c) are effectively no different to coding:

bool success = false;
for (int i = 1: i <= 5 && !success; i++)
{
    var response = apiConnector.ExecuteAndReturnResponse("/api/endpoint");
    success = r != null && r.IsSuccessStatusCode && r.Content.ReadAsAsync<IEnumerable<MyType>>().Result.Any())
    if (!success) Thread.Sleep(TimeSpan.FromSeconds(Math.Pow(2, i)));
}

Blocking on async calls is a common and widely documented source of deadlocks. If you want/need to remain with using HttpClient synchronously, you probably want to consult existing information on doing so without deadlocking, eg: 1, 2, 3

If you choose to refactor the code to use async/await throughout (which brings benefits like not blocking threads while waiting for a response; and avoiding deadlocks), then the question that Polly does not have a handle predicate of the form .HandleResultAsync<TResult>(Func<Task<TResult, bool>>) may arise again. If that becomes an issue, please do let me know, as I can happily advise several options to structure code to avoid it.

Hope that some of the links above provide useful information on the deadlocking issue!

[reisenberger]

Hi again @subatta !

To follow up on our previous conversation, if you wanted to refactor the code presented to async/await, here is an example of how it could be tested with Polly:

private async Task<bool> ResultContainsAny<MyType>(this HttpResponseMessage r)
{
    if (r == null || !r.IsSuccessStatusCode) return false;
    var readResult = await r.Content.ReadAsAsync<IEnumerable<MyType>>();
    return readResult.Any();
}

async void MyTest() // assumption: your test framework and version supports async test methods
{
    // Arrange
    var retryPolicy = Policy
        .HandleResult<bool>(false)
        .WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

    // Act
    // - whatever you do initially in the test for "messages are put on a bus"

    // Assert.
   Assert.IsTrue( // or whatever your assertion syntax
        await retryPolicy.ExecuteAsync<bool>(() => httpClient.GetAsync("/api/endpoint")
            .ContinueWith(t => t.Result.ResultContainsAny<MyType>()));
   );
}

(As discussed, this moves the await on ReadAsAsync() out of the handle clause; but it tries to keep the intent clear, with a ResultContainsAny<>() extension method.)

Obviously, a refactor to async/await may not suit all situations (it may not be 'minimally invasive'), but I hope that the example is useful!

[subatta]

Thanks, @reisenberger. My initial thought when I had deadlock issue is to have async tests which solve the issue like you outlined above. Unfortunately, we use MSTest which doesn't support async void type of tests. I guess limitations exist at several fronts on our side that need addressing.

Thanks a million for your help!

[reisenberger]

( async void a slip there on my part [due to habit of writing void-returning test methods] that should of course be avoided; but it should work just the same as an async Task test method )

[tucaz]

Hi @reisenberger! I'm sorry to revive this thread, but I'm facing a situation close to what you guys talked about and I can't get your sample to work in any way.

Here's what I'm trying to build on top of your example:

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using Backend.PayPal;
using Polly;
using Xunit;
using Xunit.Sdk;

namespace Scripts
{
    public class PollyTest
    {
        [Fact]
        async void MyTest() // assumption: your test framework and version supports async test methods
        {
            var retryPolicy = Policy
                .HandleResult<bool>(false)
                .WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));

            var client = new HttpClient();

            await retryPolicy.ExecuteAsync(() =>
                client.GetAsync("https://google.com").ContinueWith(t => t.Result.ResultContainsAny<MyType>()));
        }
    }

    public static class Foo
    {
        public static async Task<bool> ResultContainsAny<MyType>(this HttpResponseMessage r)
        {
            if (r == null || !r.IsSuccessStatusCode) return false;
            var readResult = await r.Content.ReadAsAsync<MyType>();
            return readResult != null;
        }
    }
}

The only change I would need is to actually get the HttpResponseMessage from the successfull Polly call.

Right now I'm getting:

 PollyTest.cs(24, 17): [CS0029] Cannot implicitly convert type 'System.Threading.Tasks.Task<System.Threading.Tasks.Task<bool>>' to 'System.Threading.Tasks.Task<bool>'

and

PollyTest.cs(24, 17): [CS1662] Cannot convert lambda expression to intended delegate type because some of the return types in the block are not implicitly convertible to the delegate return type

[reisenberger]

@tucaz Thanks for picking up on that.

Cannot implicitly convert type 'System.Threading.Tasks.Task<System.Threading.Tasks.Task<bool>>' to 'System.Threading.Tasks.Task<bool>'

tells us we're dealing with a nested task (Task<Task<bool>>); the recommended approach is .Unwrap(). The scenario is exactly parallel to the second example on that linked Microsoft page. So:

bool outcome = await retryPolicy.ExecuteAsync(() =>
            client.GetAsync("https://google.com").ContinueWith(t => t.Result.ResultContainsAny<MyType>()).Unwrap());

---

The only change I would need is to actually get the HttpResponseMessage from the successful Polly call

To return something else as well as the bool from the execution, you could refactor to make the return type (bool outcome, MyType myType) or (bool outcome, HttpResponseMessage message).