Tonic: concurrent use of client

[scottlamb]

Feature Request

Crates

both tonic & tonic-build

Motivation

I want to issue a bunch of concurrent API calls over the same channel. It looks like tonic doesn't support this today given that all the client methods take&mut self. The official gRPC implementations (C++, Java, Go) all support this. It's common to have heavy concurrency over the same channel. This seems much more practical than having tons of channels and managing a pool of them from the caller for at least a few reasons:

• ergonomics - it'd be extra work for the caller to do that pooling, especially once you get into building client-side load balancing on top of a bunch of channels to individual servers
• efficiency - there's RAM overhead to channels (socket buffers) and even CPU overhead (health checking). With client-side load balancing these can be significant even without throwing in multiple connections to the same server.
• limits - you could exhaust the number of local ephemeral ports and be unable to open any more channels to the same server because you've hit the fundamental TCP limit on having only one connection per (source ip, source port, destination ip, destination port). iirc, the OS also won't automatically pick an ephemeral port for you that anything else is using (even to talk to a different ip:port than you're connecting to), so that's an even more restrictive limit unless you start manually binding to ports when connecting outbound.

Proposal

API and semantics: I'd like all the generated API calls to simply take &self rather than &mut self and fire off the RPCs in parallel, subject to the server's MAX_CONCURRENT_STREAMS limit. If there are more calls than that limit, later ones block until earlier ones finish before sending. See also the max_streams test mentioned here.

Internals: I haven't looked into tonic internals at all so no opinions there.

Alternatives

I think the only reasonable alternative would be to have a cloneable client where each clone shares the underlying channel.

[LucioFranco]

@scottlamb Hey! So the reason generated clients take a &mut self has nothing really to do with not being able to support concurrent requests on the same "channel" but in fact its to follow the tower model.

So at the core tonic's client is abstracted around GrpcService which is a type alias for tower::Service that provides extra bounds to enforce that this is a http service. Tower has a concept of natural backpressure via the Service::poll_ready fn. Due to this being the abstraction our generated clients which just wrap some GrpcService must abide by the same borrow rules. This forces us to require all clients taking a &mut self.

The solution here is to actually clone the client. This should be very cheap as it is just a ref bump, basically the same as mpsc::Sender::clone(). So in reality we could take &self and clone on each request but since tonic has an abstraction enforced by tower service we can't expect all GrpcService to be clone and therefore it doesn't make sense to take a &self here.

The big key here for why tower even needs to take a &mut self is due to poll_ready being stateless and only one owner of a service can poll_ready to get a single slot available. This is based on the contract in tower service. We have found in general that this works quite well.

Performance wise we have found you can get a decent amount of throughput by just clonling a single channel. Though the best comes when you load balance a few channels and also clone them around.

Now this 100% could be documented better but I am not exactly sure what the right place might be.

Let me know if this doesn't make sense, happy to explain more :)

TL;DR: The work around is to call clone on the client and that is how you can dispatch requests concurrently.

[scottlamb]

Thanks for the explanation!

Yeah, I think there should at least be some documentation saying that clone is supposed to be cheap for the stock transport. I don't think I'm going to be the only one surprised, given that other language's gRPC implementations just have one client object you can share between threads. Maybe some/all of:

• an example/benchmark that clones a client for concurrency; if there is one already, I'm missing it.
• some reference page about generated clients
• the docstring for client::Grpc and/or GrpcService

[Sushisource]

Huge plus one there. @LucioFranco Your explanation was super valuable and pretty much a slightly cleaned up version of that thrown straight into the docs (docstring would be great too as @scottlamb suggests) would really help make this perf characteristics clear here.

I'd be happy to make a PR with doc / docstring updates if that helps

[LucioFranco]

@Sushisource I'd absolutely love a PR with that!

[LucioFranco]

We landed this PR so closing this issue!