Polly: How to use WaitAndRetry with jitter retry intervals depending on the Exception thrown

[FrancescoReviso]

I would like to apply a WaitAndRetry policy with different parameters, depending on the Exception thrown, for the jitter function calculating the retry intervals. Let's say that for Type1Exception I would like to feed to the function a specific couple of maxValue and seedValue and for Type2Exception others. A simple jitter function that calculates the retry intervals is as follows:

IEnumerable<TimeSpan> GetRetryIntervals(int maxRetries, TimeSpan seedDelay, TimeSpan maxDelay)
{
     var jitterer = new Random();
     int retries = 0;
     double seed = seedDelay.TotalMilliseconds;
     double max = maxDelay.TotalMilliseconds;
     double current = seed;

      while (++retries <= maxRetries)
      {
         if (retries == maxRetries)
     {
            yield return maxDelay;
        yield break;
         }

      current = Math.Min(max, Math.Max(seed, current * jitterer.NextDouble()));
      yield return TimeSpan.FromMilliseconds(current);
  }
}

The only way I found to use the jitter with such requirements is to retrieve both jitter intervals before creating the policy and use retryCount as index:

var type1Intervals = GetRetryIntervals(10, TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(10000));
var type2Intervals = GetRetryIntervals(10, TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(20000));

Policy
  .Handle<Exception>()
  .WaitAndRetryAsync(5, async (exception, retryCount) =>
   {
      if(exception is Type1Exception)
      {
          await Task.Delay(type1Intervals.ElementAt(retryCount)).ConfigureAwait(false);
      }
      if(exception is Type2Exception)
      {
          await Task.Delay(type2Intervals.ElementAt(retryCount)).ConfigureAwait(false);
      }
   }   
  })
  // execute the command
  .ExecuteAsync(async () => await DoSomething())
  .ConfigureAwait(false);

This solution indeed works but it is neither elegant not optimized. Is there a more proper way to implement it?

[reisenberger]

Here is an option for jittering with two different scaling and max-clamp values:

var jitterEnumerator = GetRetryIntervals(5, TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(20000)).GetEnumerator();

var retryVaryingJitterPerExceptionType = Policy
  .Handle<Type1Exception>()
  .Or<Type2Exception>()
  .WaitAndRetryAsync(5, sleepDurationProvider: (retryCount, exception, context) =>
    {
        bool isType1 = exception is Type1Exception;
        double max = isType1 ? 10000 : 20000;
        if (!jitterEnumerator.MoveNext()) return TimeSpan.FromMilliseconds(max);
        return TimeSpan.FromMilliseconds(Math.Min(max, jitterEnumerator.Current.TotalMilliseconds * (isType1 ? 2 : 1))); // 2 == 100/50 from original type1Intervals and type2Intervals 
    });

This consumes just one Enumerable and uses some constants to scale it and max-clamp it to match the numerics of the original.

---

Note that the GetRetryIntervals(...) method quoted doesn't jitter. It generates a stream of the value seed, jumping to finish at maxDelay.

You could fix that by changing the relevant line to:

current = Math.Min(max, Math.Max(seed, current * (1 + jitterer.NextDouble())));

Note that I haven't had time to stochastically analyze the distribution of that ^ simple fix to GetRetryIntervals(...). We also offer some off-the-shelf jitter formulae we recommend, in a nuget package Polly.Contrib.WaitAndRetry. These have been stochastically analyzed to offer a good combination of well-distributed jitter and exponential backoff.

[reisenberger]

A couple of technical comments about the original quoted code, in case you eventually settle on a solution more like that.

First, with .WaitAndRetryAsync(...), you don't need to manually call await Task.Delay(type1Intervals.ElementAt(retryCount)) yourself. The call to await Task.Delay(...) is already built in to the policy implementation. You only need to configure the policy to specify the delay timespans, as shown in the readme examples for WaitAndRetry.

Second, be aware that:

var type1Intervals = GetRetryIntervals(10, TimeSpan.FromMilliseconds(100), TimeSpan.FromMilliseconds(10000));

with

type1Intervals.ElementAt(retryCount)

will get a new enumerator at each call to .ElementAt(...), and thus each call to .ElementAt(retryCount) in your original code will in fact be based on a different, freshly-realised sequence of the enumerable. With the original code, you might want to add a .ToList() or .ToArray() thus:

var type1Intervals = GetRetryIntervals(...).ToArray();

to reify the sequence just once.

[FrancescoReviso]

@reisenberger thank you very much for your rich reply. Is it also possible to apply an "except logic" to the exception thrown? Suppose I would like to apply a WaitAndRetry behaviour for all the types of exceptions except one, in this case Type1Exception, will this code work?

var jitterEnumerator = GetRetryIntervals(5, TimeSpan.FromMilliseconds(50), TimeSpan.FromMilliseconds(20000)).GetEnumerator();

var retryVaryingJitterPerExceptionType = Policy
  .Handle<Exception>()
  .WaitAndRetryAsync(5, sleepDurationProvider: (retryCount, exception, context) =>
    {
        bool isType1 = exception is Type1Exception;
        double max = isType1 ? 10000 : 20000;
        if (!jitterEnumerator.MoveNext()) return TimeSpan.FromMilliseconds(max);
        return TimeSpan.FromMilliseconds(Math.Min(max, jitterEnumerator.Current.TotalMilliseconds * (isType1 ? 2 : 1))); // 2 == 100/50 from original type1Intervals
    });

[reisenberger]

@FrancescoReviso The code in the comment above will apply the "type 1" WaitAndRetry behaviour to any exception is Type1Exception, and the other WaitAndRetry behaviour to all other exceptions.

Re:

Is it also possible to apply an "except logic" to the exception thrown?
Suppose I would like to apply a WaitAndRetry behaviour for all the types of exceptions except one, in this case Type1Exception

If you want the WaitAndRetry policy to _not apply at all_ for one exception type, you can make a policy to handle all exceptions except that, by using a single handle clause like this:

.Handle<Exception>(e => !(e is Type1Exception))

[reisenberger]

Closing due to no further questions recently raised. If you do have further questions, please post again and we can re-open.