Grpc-java: Retries as a first-class feature

[ejona86]

Easy-to-enable, first-class API for having automatic retry any type of call. It would be good if this API supported hedging, where you preemptively issue a second RPC if the first is taking a while. Supporting hedging means that hedging would not need additional buffering.

This is larger in scope than #1122, which is only unary. This is related to #1570 and #1586, but more concerned with the API to expose to applications and the cross-language feature set. I've not yet looked at the policy in those PRs, so this says nothing about how I feel about the policy presented there. We can end up copying some of the policy in the PRs into this discussion.

[ejona86]

I spoke with @ctiller and have this rough design that could suit both Java and C core.

I think any "policy" should probably be set on CallOptions (with maybe a default specified in ManagedChannelBuilder).

The design here is based on the same basic approach taken with the Load Balancer, have an object that can direct the core what to do, which seems to have worked well. The proposal is to have something like (very pseudo-code; just intended to describe features available, not concrete methods/arguments):

RetryPolicy {
  int maxBytesToBuffer;
  void start(Listener, method);
  void inboundProgress(headers|message|trailers);
  void fail(how);

  Listener {
    void event(pick|end);
  }
}

(There is likely a factory to create a RetryPolicy instance each request; not too concerned at this point.) When a request is started, start() would be called and provided an object to provide later (possibly async) events. "pick" instructs the runtime to choose a transport and start the stream on that transport. "end" means there will be no further retry events.

The runtime will buffer outgoing messages up to maxBytesToBuffer (rounded up to the nearest message), until the "end" event. When the too much is buffered, flow control will push-back on the application.

You can see we have inboundProgress(), but no outboundProgress(). That's not a design feature; we were okay with having an outboundProgress() if it proved useful. We do however, want to separate the internal "streams" from this API, for the same reasons why "transports" aren't exposed concretely to LoadBalancers.

Noop Retry, just issues the request like normal:

void start(Listener, method) {
  listener.event(pick);
  listener.event(end);
}

Retry once:

void start(Listener, method) {
  this.listener = listener;
  listener.event(pick);
}

void fail(how) {
  if (!retried && retryable(how)) {
    retried = true;
    schedule(1, SECONDS, lambda: listener.event(pick));
  } else {
    listener.event(end);
  }
}

Hedging:

void start(Listener, method) {
  this.listener = listener;
  listener.event(pick);
  // There would be some way to pass attributes to the LB so the pick would be
  // guaranteed to go to a different transport.
  scheduled = schedule(10, MILLISECONDS, lambda: listener.event(pick));
}

void inboundProgress(headers|message|trailers) {
  scheduled.cancel();
  listener.event(end);
}

[ejona86]

@lukaszx0, @elandau, FYI

[elandau]

@ejona86 I like this design. Regarding the factory aspect how about returning a RetryContext from start()? I think it would make it clearer as to what tracks state.

RetryPolicy {
  RetryContext start(Listener, method);

  Listener {
    void event(pick|end);
  }

  RetryContext {
     void inboundProgress(headers|message|trailers);
     void fail(how);
  }
}

[ejona86]

@elandau, I think returning the context from start is a good idea, since it would allow using final/constructor. Other places we can't do that (like Call, Stream, *Transport), but I do think we can do so here. We may want to tweak the name start(), but that is bike shedding that can be done later.

[lukaszx0]

Thanks for putting this together Eric.

I've not yet looked at the policy in those PRs

FYI, in our PR (https://github.com/grpc/grpc-java/pull/1570), there's RetryPolicy which looks like this:

public class IdempotentRetryPolicy extends RetryPolicy {
  public static final class Provider implements RetryPolicy.Provider {
    @Override
    public RetryPolicy get() {
      return new IdempotentRetryPolicy();
    }
  }
  @Override
  public boolean isRetryable(Status status, MethodDescriptor method, CallOptions callOptions) {
    return method.isIdempotent();
  }
}

(with maybe a default specified in ManagedChannelBuilder).

It is now possible to do this after my changes in https://github.com/grpc/grpc-java/pull/1587

[lukaszx0]

cc @jhump @ericzundel

[lukaszx0]

Quick update. There's now an RFC design for retires which was aproved and I'm hoping to be the one who'll take on implementing it in my free time.

I've met with @ejona86 and we've discussed the java specific designs. We took some nodes, they're very raw, but I thought it'd be good to share them so that we can continue brainstorming and iterating on the design here.

Buffering

Two fundamental options for what to buffer:

Messages (InputStream)

• Natural at ClientCallImpl
• Work with InProcessTransport without issue

Problems:

• Re-serializes protobufs on retry

Save a byte[] copy

Problems:

• don’t expect this is acceptable; hurts memory usage and performance due to additional copies

WritableBuffer

• Add a new method on the ClientTransportFactory to get the allocator so that ClientCallImpl can pre-serialize and save the buffers

Problems:

• Hinders small message batching to use one buffer for many messages.
• Solution to that, is to save output of MessageFramer; which makes it harder to do in ClientCallImpl. Maybe could make use of MessageFramer interface being added in https://github.com/grpc/grpc-java/pull/2839

Where to implement it?

• ClientCallImpl - Seems the most natural
• AbstractStream - Sees the serialized output
• Stream “interceptor”

Open Questions

• Do we want InProcessTransport to work? How useful?
• Is re-serializing on retry acceptable for hedging?
• What is the threading going to look like?

I was planning to spend some time playing with it to get some proof of concept.

@ejona86 have you thought about it more? Got anything to add/update, comment?

[ggarber]

What is the status of this feature? Still planned to be implemented?

[ejona86]

@ggarber, yes, still planned for implementation, based on the gRFC. It's still basically in design stage for Java.

[kraghu]

@ejona86 I see retries are in design phase. How do i configure retries in grpc android ? Is there a way to see the configure retries. I see right now grpc is not attempting to retry even when the network is back and available .

[ejona86]

@kraghu, retries aren't implemented, so there isn't any way to turn them on. Reconnecting _is_ implemented though, mostly following the connection backoff spec. You may be interested in #2169.

[lukaszx0]

For the record, there has been some activity in this area recently: #3656, #3818, #3840

[dapengzhang0]

Tracking progress in the project Retry Support

[yuzheng21]

Hi, this is a very nice feature. When will this be ready in release?

[dapengzhang0]

@mackeyzheng It is not ready right now because service config is not usable. This will soon be fixed and hopefully retry will be available in release v1.12.

[stef2georg]

Hi, how is the progress on the retry mechanism going? It will simplify the client code (StreamObserver) a lot as of now our clients need to handle Status.UNAVAILABLE and resubscribe manually. Is there something that blocks the progress? I will be happy to help/contribute if additional work needs to be done.

[dapengzhang0]

Library level retry feature is available in the master branch and next release (v1.17.0). In v1.16, it is also available but there is a constraint that users must manage to disable census before enabling retry.