Polly: Is it possible to chain Fallback and CircuitBreaker? [How to construct a Failover policy]

[sledoux]

Our use case would be to have 1 circuitbreaker per fallback option, having many fallback option. The number of fallback would be dynamic. The goal being to stop retrying one of the fallback for a small period of time in case of failure. Therefore going to the next fallback automatically if the circuit of the first option is open and so on with the next fallback options.

Another question, is it possible to pass a context between the policy? For example, I would like a method to be called with some parameters. I would add the method and parameters to the context of the first policy with the fallback method and it's corresponding parameter for the next policy to call. I don't see how I could do that with Polly.

Looking at the samples and the code I don't how I could achieve those.

Any tips?

[reisenberger]

Hi @sledoux , thanks for joining the Polly conversation!

Taking first this question:

is it possible to pass a context between the policy?

Yes. Polly's Context object is intended for this:

• Every .Execute(...) (and similar) overload exists in a variant where the executed delegate can take Context as an input parameter.
• Likewise, every policy-_control_ delegate (like onRetry, onBreak etc), should also exist in a variant where that control delegate can take Context as an input parameter.

Therefore, you should be able to pass contextual information among any parts of a Polly policy, and downstream to further Polly policies part of the same execution (whether you're combining policies in some custom fashion, or automatically with PolicyWrap).

This blog post Putting the Context into Polly gives some examples.

Let us know if that's making sense, or if we can help any further on this.

[reisenberger]

Hi again @sledoux ! Re:

Our use case would be to have 1 circuitbreaker per fallback option, having many fallback option. The number of fallback would be dynamic. The goal being to stop retrying one of the fallback for a small period of time in case of failure. Therefore going to the next fallback automatically if the circuit of the first option is open and so on with the next fallback options

@aliostad Raised essentially the same q in #199, and I provided some suggestions. ( /cc @aliostad Any comments, on the final approach you took with this? )

@sledoux Two further comments, looking back at my notes to @aliostad:

• Probably recommend the approach of wrapping with a while loop or retry policy, rather than a chain of FallbackPolicys. FallbackPolicy is essentially linear, and feels like it'd be hard (or harder, while possible) to get the concept of a dynamically-extending failover, and a continuous-looping failover, using FallbackPolicy .
• The code example in that previous comment obviously has to create a closure over the variable sourceToUse. This closure can now be avoided using precisely the Context techniques discussed in comment above / related blog post.

Again, let us know how you get on with this!

[reisenberger]

@aliostad @sledoux After aliostad's original request, I wrote a proposal for a proper FailoverPolicy, aiming to generalise and encapsulate as much as possible, by targeting an IEnumerable<TProvider> .

Would love feedback on that proposal - good? bad? questions? ideas?

[sledoux]

Hi @reisenberger,

Beforehand, thank you for your answers.

Reading your first comment, I don't see a method signature to pass a context to the FallBackAction of a FallBack policy. Since the FallbackAction cannot take a parametized Func or a context, it's useless to combine it with a CircuitBreakerPolicy through a PolicyWrap, which would have been very nice. The idea would be to handle the CircuitBreakerOpen exception at the fallback policy, only switching to fallbacks when the circuit is open. This would avoid a spam on endpoints which are down for a long period of time, but scaled horizontally with multiple fallbacks for a good resilience.

The retry does have a context on the action called though so I could use that to act like a fallback with a retry count of 1. The hard part is to make this dynamically extending like you mentionned, although possible.

For now, the option that would fit our need would be to create one CircuitBreakerPolicy per item we have in a list. Wrapping that in a retry policy and using the onbreak call to cycle through the list of item like you mentioned in #199.

Looking at your proposal for the FailoverPolicy (https://github.com/App-vNext/Polly/wiki/Polly-Roadmap#failover-policy), this looks exactly like what we need, combining that with circuit breakers! We would prefer that approach over the previously mentioned one by far. Also, the possibility to manually fail could be very useful.

Hope to see this soon in a release of Polly.

[reisenberger]

I don't see a method signature to pass a context to the FallBackAction of a FallBack policy. Since the FallbackAction cannot take a parametized Func or a context, it's useless to combine it with a CircuitBreakerPolicy [with context] through a PolicyWrap

Hi @sledoux You're absolutely right, it looks like this delegate was missed among the work for v5.1. Thanks for highlighting.

This would be pretty quick to fix up (see #265), if you are interested in submitting a PR before I get to it.

[sledoux]

@reisenberger I might give it a try soon, when I get time.

[reisenberger]

@sledoux Perfect. Just post on the issue if you pick it up. I'll ditto if ditto (to avoid overlap)

[reisenberger]

@sledoux FallbackPolicy with fallbackAction taking a Context input parameter, released as part of v5.3.0

[reisenberger]

Closing due to lack of further activity. Anyone feel free to re-open to continue the discussion, tho.

[jonathann92]

Hi Sorry to bring this thread back to life.

I have multiple methods in an interface IMath with int Add() and double Divide() methods and 3 classes that implement IMath -> MathA, MathB, MathC, with the order being MathA, then MathB, then MathC.

Right now I have 2 policies, one for Add() and one for Divide(). Would it be possible to only make 1 fallback per implementation of IMath and have it failback on what I'm trying to do in the .Execute() call?

var fallbackMathBAdd = Policy<int>
                    .Handle<Exception>()
                    .Fallback(fallbackAction: (context, token) =>
                        {
                            var a = (int) context["a"];
                            var b = (int) context["b"];
                            return mathB.Add(a, b);
                        },
                        onFallback: (result, context) => { }
                    );

                var fallbackMathBDivide = Policy<int>
                    .Handle<Exception>()
                    .Fallback(fallbackAction: (context, token) =>
                        {
                            var a = (int) context["a"];
                            var b = (int) context["b"];
                            return mathB.Divide(a, b);
                        },
                        onFallback: (result, context) => { }
                    );

                var fallbackMathCAdd = Policy<int>
                    .Handle<Exception>()
                    .Fallback(fallbackAction: (context, token) =>
                        {
                            var a = (int) context["a"];
                            var b = (int) context["b"];
                            return mathC.Add(a, b);
                        },
                        onFallback: (result, context) => { }
                    )
                    .Wrap(fallbackMathBAdd);

                var fallbackMathCDivide = Policy<int>
                    .Handle<Exception>()
                    .Fallback(fallbackAction: (context, token) =>
                        {
                            var a = (int) context["a"];
                            var b = (int) context["b"];
                            return mathC.Divide(a, b);
                        },
                        onFallback: (result, context) => { }
                    )
                    .Wrap(fallbackMathBDivide);

[reisenberger]

@jonathann92 No problem. Re:

Would it be possible to only make 1 fallback [...] and have it failback on what I'm trying to do in the .Execute() call?

Some options;

• The fallbackAction takes, as you can see, a context input parameter. So you could inscribe what you are about to execute, on the Context, before placing the .Execute() call.
• Overloads of FallbackPolicy exist taking the Exception as an input parameter. So you could engineer the executed operations to signal via throwing different exceptions, or setting properties on the exception thrown.

[reisenberger]

@jonathann92 Sorry, re:

Overloads of FallbackPolicy exist taking the Exception as an input parameter. So you could engineer the executed operations to signal via throwing different exceptions, or setting properties on the exception thrown.

I quoted the wrong config overload. Since you are obviously using Policy<int>, this is the relevant overload. The input parameter DelegateResult<TResult> outcome to fallbackAction contains the thrown exception as outcome.Exception.

[jonathann92]

@reisenberger I would like to only make 1 fallback policy that would handle both int and double return types, since Add() returns int and Divide() returns a double. I'm not seeing how that overload would handle that. Could you give me an example of code so that the fallback would execute either Add() or Divide() depending on what was used in the Execute() method?

[reisenberger]

Thanks @jonathann92 . Re:

I would like to only make 1 fallback policy that would handle both int and double return types, since Add() returns int and Divide() returns a double

Ok, I did not understand that from the original question since all the policies were declared Policy<int>. I missed it in the opening comment about int Add() and double Divide() - thanks for the clarification.

Short answer: this is not possible (++). Polly policies governing a return type are generic in a single type TResult; they can't be generic in two different types at once. This is no different from any other generic, eg the fact that you can't make a single instance of List<T> which is both a List<int> and List<double> at the same time.

Unless of course you define the generic in some common base type. In this case, your only such choice is object. So you could define, eg Policy<object>.Handle<Exception>.Fallback(/* etc */). This of course will incur some boxing/unboxing, as int and double are structs which will get boxed to object, but that boxing may not matter unless you have high-performance requirements.

Could you give me an example of code so that the fallback would execute either Add() or Divide() depending on what was used in the Execute() method?

The policy cannot know what was used in the Execute() method in some general sense. Per my previous comment, either you have to signal manually what you execute by setting some value on context; or signal what you did execute and which failed by attaching some information to the thrown Exception.

I can give examples, but it would be useful to know first if this is still useful to you, after my statement tagged (++) above.

Could you also give some more description/background on what the original problem is, that this code is targeting? It feels like we are starting to jump through some very complicated hoops, so I am asking in case there may be some simpler way of approaching the problem. Thank you.

[jonathann92]

@reisenberger

Thanks for the reply. I have fallen victim to copy and paste code lol

I think what I am trying to achieve is beyond the fallback policy’s scope. I wanted to make a chain of failover implementations of the IMath interface.

For example they all make an http call to some api to do their math operations. I wanted to call ‘MathA’ first but if it fails it would use ‘MathB’ instead and so on. I want to only configure one fallback implementation that would apply to all methods exposed by ‘IMath’. Also each implementation would have a circuit breaker on it but that’s a separate thing.

[reisenberger]

Thanks @jonathann92 . Is your real-world goal a system which fails over to a new server/provider amongst a server/provider-set, when the 'current' targeted server fails some/too many operation/s? If so, see this thread and answer. That example cycles to the next server on any retry, in onRetry. You could alternatively trigger the failover on the _n_-th retry, or when the circuit for a downstream instance breaks (using onBreak in the circuit-breaker policy). Using retry or retry-with-circuit-breaker to trigger failover is probably a more natural fit for this goal than a chain of fallback policies.

In such a setup, you would likely want a circuit-breaker per downstream-instance.

Retry and circuit-breaker policies used in this pattern can be non-generic (if you don't depend on the return result to decide whether to break or retry), so that they don't fall into the struggle you are having with trying to marry a strongly-typed FallbackPolicy<T> with methods on an interface IFoo returning different types.

[jonathann92]

@reisenberger

Thanks for the references, those are really helpful and creative ways of using a different source onBreak.

My real-world goal is similar to what you described but I can't use fallback servers because each server is made by different people so their queries are different. Like for example MathA might be calling a REST api and MathB could be calling a SOAP api. In this case I would like to fallback on different implementations.

In the examples you provided I could put all IMath implementations into an array with a int currentMathImplementationIndex = 0 and onBreak I would increment the currentMathImplementationIndex by 1 with % as you suggested. However I have a priority ordering of which IMath implementation to use. So if MathA's circuit is reset I would like to use MathA instead of MathB or MathC

A reason for this is that when the Divide() method is called the different providers can return different precisions. Ideally it would be nice if they all behaved the same.
A second reason is that MathA is cheaper per call than MathB and MathB is cheaper than MathC.

I saw in this same issue you mentioned a proposal for a failvoer policy that took in an IEnumerable<TProvider> in 2017. I think thats why my search led me to this issue. Did the proposal ever get implemented?

[reisenberger]

@jonathann92 FailoverPolicy never rose to the top of the priority tree for implementation. There are also some conceptual challenges with implementing it for Polly: Polly has no .Execute<TInput, TResult>(...) overloads for the any-TInput case; and adding those (or something similar so that a FailoverPolicy could sit in the middle of a PolicyWrap) is, for reasons too long to explain, a reasonably significant rewrite of Polly.

If a library like Polly isn't suiting your needs, it's probably simplest to code your own thing. If I was doing this outside Polly (free from trying to fit it into PolicyWrap), I would probably just code _something like_:

    // deliberately long method name for explanation
    public static TResult ExecuteTryingImplementationsInPreferredOrderUntilSuccessOrAllFail<TInput, TResult>(this IEnumerable<TInput> implementations, Func<TInput, TResult> operation)
    {
        Exception lastException = null;
        foreach (TInput implementation in implementations)
        {
            try
            {
                // If you want some Polly policies like circuit-breakers around the operation, you can execute the line below through policies.
                return operation(implementation);
            }
            catch (Exception e)
            {
                lastException = e;
                // Do some logging here too if you want.
            }
        }

        throw lastException; // If all implementations fail, it will throw. So you want the caller to defend against failure of all implementations; or implement some retry _outside_ this method for that case; or log; whatever.
    }

To always prefer MathA over MathB over MathC after any failure, in case the circuit for MathA (etc) has reset again, one could use an int index and just reset to index = 0 in the circumstances you want to try from the 'most preferred' implementations downwards again:

_EDIT: This implementation keeps trying until it exhausts all of MathA, MathB and MathC into broken circuits - maybe that's not what you want, but it should be easy to extrapolate from._

    // deliberately long method name for explanation
    public static TResult ExecuteRepeatedlyPreferringEarlierImplementationsProvidedCircuitsNotBrokenUnlessAllCircuitsBroken<TInput, TResult>(this IEnumerable<TInput> implementations, Func<TInput, TResult> operation)
    {
        Exception lastException = null;

        IList<TInput> impls = implementations.ToList();
        int implCount = impls.Count;

        for (int index = 0; index < implCount; )
        {
            try
            {
                // for some previously-created circuitBreakers[] collection of a circuit-breaker per implementation - might need passing in to the method
                return circuitBreakers[index].Execute(() => operation(impls[index])); // Check you don't have the 'access to modified closure' problem and work round with the usual technique of copying the 'index' variable if necessary. I haven't checked.
            }
            catch (BrokenCircuitException bce)
            {
                lastException = bce;
                // Log it also if you want.

                // This kind of exception means we should try the next implementation.
                index++;
            }
            catch (Exception e)
            {
                // Log it if you want.

                // Start from the beginning again: those at the start of the implementations are preferred - unless their circuits are broken.
                index = 0;
            }
        }

        throw lastException; // If all implementations fail, it will throw. So you want the caller to defend against failure of all implementations; or implement some retry _outside_ this method for that case; or log; whatever.
    }

Leading to simple client code, eg:

// for some List<IMath> implementations
var result = implementations.Execute...(math => math.Divide(1, 2));

(Just some sketches - you might need or prefer something different.)

_EDIT_: These suggestions try each implementation in sequence as this corresponds to your problem statement. Of course, you could also proceed in parallel with fan-out/fan-in if that suited.

[jonathann92]

Thanks @reisenberger this helped a lot!