Grpc-dotnet: Keep alive missing?

[TheLever]

For some reason my question text was removed from the question - redoing it here.

Is there a way to keep the connection between client and server alive in a C# gRPC app? My scenario is a bidirectional streaming one (essentially the FullStockTicker demo). What I see is that the server closes the HTTP/2 connection after a few minutes. I have tried to find a way to set the keep alive flag, but that does not seem possible from C#.

I am using Grpc.AspNetCore (2.27.0), Grpc.Tools (2.27.0), Google.Protobuf (3.11.3) and generated clients.

[anurse]

Can you provide a runnable sample that reproduces this problem? We don't believe the server should be terminating the connection in this situation, so it would help to see the full context of what you're doing.

In addition, if you could enable more detailed logging using the following config in your appSettings.json that would help.

{
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "System": "Information",
      "Microsoft": "Trace",
      "Grpc": "Debug"
    }
  }
}

[halter73]

@ndglover Can you provide the logs that @anurse asked for? Thanks.

[thefringeninja]

This is something we will definitely need, as we expect our subscriptions to run indefinitely.

[JuliusSweetland]

@JamesNK @anurse Hi James, Andrew, I believe I might need keep alive functionality where I am implementing bidi and server to client streams. These would be long running, and there may be long periods of no messages, which is what I understand the keep alive ping/pong behaviour to be designed for - essentially to test that the connection has not been reset without either side knowing. Would this be a suitable use case for keep alive? If so I'd like to throw my vote in for including it with Grpc-net. Thanks.

[JuliusSweetland]

If it helps there is one C# project on GitHub which seems to have rolled their own grpc client code which allows them to set channeloptions on the underlying channel:

https://github.com/zeebe-io/zeebe-client-csharp/blob/5d416bc1bb6eccc3898f0727f639ecd1b2bfb434/Client/ZeebeClient.cs#L58

Which uses this GatewayGrpc client:

https://github.com/zeebe-io/zeebe-client-csharp/blob/5d416bc1bb6eccc3898f0727f639ecd1b2bfb434/Client/Impl/proto/GatewayGrpc.cs

[JuliusSweetland]

@JamesNK @anurse Hi again James, Andrew, can I ask where support for gRPC keepalive might be on your priority list? At present I am rolling my own heart beating, which is very much sub-optimal. I would love to rip out some of my code and use the underlying mechanism to get confidence that longstanding tcp connections are still connected. Thanks.

[JamesNK]

At the moment it is low priority because we haven't had many people ask for it. What do you want built-in heart beats for? Is it perf, i.e. keeping the TCP connection open so there is not a delay reestablishing the connection after a pause?

[JuliusSweetland]

A combination of requiring long running streams for performance reasons, but I also require an up to date view of which streaming connections are still live as we require a client to have connections to certain streams before we allow them to perform operations against other endpoints. If we cannot guarantee the client connections are currently active then we cannot enforce policies which we require.

[MaxXor]

@JamesNK I'm planning to switch to gRPC but this issue kept me back. My project has long-running idle connections and need to be responsive as soon as they are used.

[vadimi]

We are using keepalives in grpc core when exposing grpc services through AWS NLB load balancer, without them the load balancer would just non gracefully close tcp connections and grpc clients would get errors. My understanding is Azure load balancer works the same way. When running services in kubernetes it also has idle timeout settings (https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) which will non gracefully terminate tcp connections as well. Keepalives really help in these scenarios which makes apps and services a lot more stable.

[JuliusSweetland]

@JamesNK Hi James - has any of the above feedback influenced how you are thinking about Keep Alive?

[JamesNK]

I'm planning to investigate what options are in Grpc.Core, and what ones would be useful to have in grpc-dotnet.

If people can give feedback on which settings are important, and why, then that would be useful.

To set expectations: Adding new features to the gRPC on the server - including Kestrel - will be easier than the client. I can prioritize implementing server-side features myself. Client-side features that need to be added to HttpClient are done by a different team. They might not make the cut in the .NET 5 time-frame.

[chwarr]

Below I've included what my product is currently using to keep Azure load balancers alive when we have a lull in traffic and our gRPC channels don't have any active calls or messages. (The Azure load balancers will sometimes drop idle connections without sending RST packets to either endpoint.)

We use keepalives to

1. reduce latency by having a connection to our backend already established (don't have to do TCP, TLS, gRPC, &c. handshakes while in the middle of a client's request) and
2. improve reliability by detecting when connections die during idle times.

As has been mentioned before, yes, we could do this by adding some sort of heartbeat in our application protocol. The keepalive functionality in the gRPC code library has, thus far, been sufficient.

This is an excerpt of the keepalive options that are set on a gRPC code channel.

The ExtendedChannelOptions class is a collection of C# string constants for the gRPC code channel options. It's definition is below.

// This channel argument controls the period (in milliseconds) after which a
// keepalive ping is sent on the transport.
new ChannelOption(ExtendedChannelOptions.KeepAliveTimeMilliseconds, 2500),

// This channel argument controls the amount of time (in milliseconds), the sender of
// the keepalive ping waits for an acknowledgment. If it does not receive an
// acknowledgment within this time, it will close the connection.
new ChannelOption(ExtendedChannelOptions.KeepAliveTimeoutMilliseconds, 60000),

// This channel argument if set to 1 (0 : false; 1 : true), allows keepalive pings to be sent even if there are no calls in flight.
new ChannelOption(ExtendedChannelOptions.KeepAlivePermitWithoutCalls, 1),

// This channel argument controls the maximum number of pings that can be sent when
// there is no other data (data frame or header frame) to be sent. GRPC Core will not
// continue sending pings if we run over the limit. Setting it to 0 allows sending
// pings without sending data.
new ChannelOption(ExtendedChannelOptions.MaxPingsWithoutData, 0),

// If there is no data being sent on the transport, this channel argument controls
// the minimum time (in milliseconds) gRPC Core will wait between successive pings.
new ChannelOption(ExtendedChannelOptions.MinTimeBetweenPingsMilliseconds, 1000),

ExtendedChannelOptions definition

public static class ExtendedChannelOptions
{
    /// <summary>
    /// This channel argument controls the period (in milliseconds) after
    /// which a keepalive ping is sent on the transport.
    /// </summary>
    public const string KeepAliveTimeMilliseconds = "grpc.keepalive_time_ms";

    /// <summary>
    /// This channel argument controls the amount of time (in
    /// milliseconds), the sender of the keepalive ping waits for an
    /// acknowledgement. If it does not receive an acknowledgement within
    /// this time, it will close the connection.
    /// </summary>
    public const string KeepAliveTimeoutMilliseconds = "grpc.keepalive_timeout_ms";

    /// <summary>
    /// This channel argument if set to 1 (0 : false; 1 : true), allows
    /// keepalive pings to be sent even if there are no calls in flight.
    /// </summary>
    public const string KeepAlivePermitWithoutCalls = "grpc.keepalive_permit_without_calls";

    /// <summary>
    /// The load balancer policy that should be used.
    /// </summary>
    public const string LoadBalancerPolicyName = "grpc.lb_policy_name";

    /// <summary>
    /// This channel argument controls the maximum number of pings that can
    /// be sent when there is no other data (data frame or header frame) to
    /// be sent. GRPC Core will not continue sending pings if we run over
    /// the limit. Setting it to 0 allows sending pings without sending
    /// data.
    /// </summary>
    public const string MaxPingsWithoutData = "grpc.http2.max_pings_without_data";

    /// <summary>
    /// If there is no data being sent on the transport, this channel
    /// argument controls the minimum time (in milliseconds) gRPC Core will
    /// wait between successive pings.
    /// </summary>
    public const string MinTimeBetweenPingsMilliseconds = "grpc.http2.min_time_between_pings_ms";

    /// <summary>
    /// If there is no data being sent on the transport, this channel
    /// argument on the server side controls the minimum time (in
    /// milliseconds) that gRPC Core would expect between receiving
    /// successive pings. If the time between successive pings is less that
    /// than this time, then the ping will be considered a bad ping from
    /// the peer. Such a ping counts as a ‘ping strike’. On the client
    /// side, this does not have any effect.
    /// </summary>
    public const string MinPingIntervalWithoutDataMilliseconds = "grpc.http2.min_ping_interval_without_data_ms";

    /// <summary>
    /// This arg controls the maximum number of bad pings that the server
    /// will tolerate before sending an HTTP2 GOAWAY frame and closing the
    /// transport. Setting it to 0 allows the server to accept any number
    /// of bad pings.
    /// </summary>
    public const string MaxPingStrikes = "grpc.http2.max_ping_strikes";

    /// <summary>
    /// Maximum time that a channel may have no outstanding rpcs. Int valued,
    /// milliseconds.INT_MAX means unlimited.
    /// </summary>
    public const string MaxConnectionIdleMilliseconds = "grpc.max_connection_idle_ms";
}

[JuliusSweetland]

@JamesNK As detailed in the previous post. For me the most useful would be the first 3 options:

/// <summary>
/// This channel argument controls the period (in milliseconds) after
/// which a keepalive ping is sent on the transport.
/// </summary>
public const string KeepAliveTimeMilliseconds = "grpc.keepalive_time_ms";

/// <summary>
/// This channel argument controls the amount of time (in
/// milliseconds), the sender of the keepalive ping waits for an
/// acknowledgement. If it does not receive an acknowledgement within
/// this time, it will close the connection.
/// </summary>
public const string KeepAliveTimeoutMilliseconds = "grpc.keepalive_timeout_ms";

/// <summary>
/// This channel argument if set to 1 (0 : false; 1 : true), allows
/// keepalive pings to be sent even if there are no calls in flight.
/// </summary>
public const string KeepAlivePermitWithoutCalls = "grpc.keepalive_permit_without_calls";

[vadimi]

@JamesNK here are the options that we normally use for keepalives on the client:

var opts = new List<ChannelOption>();
opts.Add(new ChannelOption("grpc.keepalive_time_ms", connOptions.KeepaliveTimeout));
opts.Add(new ChannelOption("grpc.keepalive_timeout_ms", ConnectionOptions.DefaultKeepaliveWaitTimeout));
opts.Add(new ChannelOption("grpc.keepalive_permit_without_calls", 1));
opts.Add(new ChannelOption("grpc.http2.min_time_between_pings_ms", connOptions.TimeBetweenPings));
opts.Add(new ChannelOption("grpc.http2.max_pings_without_data", 0));

on the server this one is useful to disable keepalive policy if needed:

new ChannelOption("grpc.http2.max_ping_strikes", 0)

not strictly related to keepalives, but we also use grpc.max_connection_idle_ms and grpc.max_connection_age_ms

[JamesNK]

Is the typical pattern here that the client is responsible for keeping the connection alive? i.e. the client pings the server on an interval and the server is only responsible for replying to the ping

[chwarr]

We run the pings in both directions so that both client and server can detect when traffic has disappeared into a black hole during idle times. (AFAIK, there's no setting to anticipate pings and update connectivity state if one hasn't been seen in a while.)

For us, client pinging server would be more useful than server pinging clients. We do have bi-directional streaming, but the idle times are usually between calls, not while a call is active. (E.g., my service doesn't have long-lived calls pushing data to clients. Yet. 😉)

[JamesNK]

I briefly looked into Grpc.Core's keep-alive pings a few months ago. The packets in Wireshark from pings indicated that they were connection level pings. The pings had a StreamID=0. I didn't see any stream level pings where the ping's StreamID=ActiveStreamID.

Do pings serve any purpose when it comes to keeping streams alive? I am thinking about potentially long running calls on those streams like bi-directional streaming that might not actively be sending messages.

[chwarr]

AFAIK, it just keeps the TCP, TLS, and HTTP2 connection alive. (My product doesn't have long-lived streams.) To control stream/call lifetime, I think one needs to use deadlines.

[JamesNK]

Ok.

Envoy supports timing out individual streams - see stream_idle_timeout at https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/network/http_connection_manager/v2/http_connection_manager.proto

I think to keep a long-running streaming call open in an environment where stream_idle_timeout is enabled you would need to send occasional dummy keepalive messages through gRPC.

[JamesNK]

Proposal for APIs added to HttpClient - https://github.com/dotnet/runtime/issues/31198#issuecomment-635273974

These could then be set when creating a handler for a gRPC channel.

[JuliusSweetland]

@JamesNK "Is the typical pattern here that the client is responsible for keeping the connection alive?"

For me it's server as well. My server maintains client subscriptions and does work for each subscriber, so knowing when the connection is dropped and cleanup can occur is useful.

[JamesNK]

Keep alive doesn't tell you when a connection is dropped. Its purpose it to send regular pings so connections are not closed from inactivity.

[thefringeninja]

But if you don't get a ping back surely that means the connection was
dropped? Unless the act of sending a ping re-establishes the connection.

On Fri, May 29, 2020 at 12:17 AM James Newton-King [e-mail]
wrote:

Keep alive doesn't tell you when a connection is dropped. Its purpose it
to send regular pings so connections are not closed from inactivity.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-dotnet/issues/770#issuecomment-635637094,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADY7B6LSDGQAOUXTKKEAQTRT3PC5ANCNFSM4KUSTWWA
.

--
Sent from my regular computer
http://twitter.com/thefringeninja
http://www.thefringeninja.com/

[JamesNK]

But if you don't get a ping back surely that means the connection was dropped?

If the client or server doesn't get a ping back in time then it means there is a problem and the connection will be dropped. What I meant is there is no API as part of this feature that will tell you a ping timed out and connection was dropped.

[JuliusSweetland]

If you have a server streaming endpoint and the client has keep alive
configured, should the connection become unusable, then wouldn't the
subsequent failed keep alive attempt from the client result in something
detectable in the client, such as an exception, or the termination of the
response stream?

On Thu, 28 May 2020, 23:30 James Newton-King, [e-mail]
wrote:

But if you don't get a ping back surely that means the connection was
dropped?

If the client or server doesn't get a ping back in time then it means
there is a problem and the connection will be dropped. What I meant is
there is no API as part of this feature that will tell you a ping timed out
and connection was dropped.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-dotnet/issues/770#issuecomment-635642362,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEA7AOMA3KRQZ7X5AWO5OTRT3QYPANCNFSM4KUSTWWA
.

[JamesNK]

If you were awaiting the response stream for more data and the connection was terminated then you would immediately get an error that the connection has closed.

[JuliusSweetland]

How does that work? If the client is waiting to receive data can it not be
waiting forever if something like an intermediate router fails? The server
at some point tries to send data, fails, and disconnects, but the client
would wait on the half open connection?

This is one of the scenarios I wanted to solve with keep alive: the client
instead regularly tests connectivity and fails faster if the connection has
become half open.

On Fri, 29 May 2020, 00:24 James Newton-King, [e-mail]
wrote:

If you were awaiting the response stream for more data and the connection
was terminated then you would immediately get an error that the connection
has closed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-dotnet/issues/770#issuecomment-635668391,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEA7ALSPRT5FFFAZ56TR2DRT3XCXANCNFSM4KUSTWWA
.

[JamesNK]

Operations awaiting HttpClient's request and response stream will error if the TCP connection is broken.

[JuliusSweetland]

Can you point me in the right direction to read about this? I don't
understand how this is possible during periods of inactivity, unless there
is some sort of packet to check whether the peer can be reached. This was
what I believed keep alive packets would probe. Thanks.

On Fri, 29 May 2020, 01:19 James Newton-King, [e-mail]
wrote:

Operations awaiting HttpClient's request and response stream will error if
the TCP connection is broken.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-dotnet/issues/770#issuecomment-635685019,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEA7ALEBMOGWZYJRYGZ54DRT35SNANCNFSM4KUSTWWA
.

[JuliusSweetland]

@JamesNK To clarify the above question: assume a streaming server. The client is awaiting the next response when an intermediary router is powered down and the connection is left half open. The server detects this and closes, but what happens with the client? What stops the client from sitting and waiting forever?

[halter73]

@JuliusSweetland You're right that it's possible for one side of an idle TCP connection to disconnect without the other side observing it. Without any keep alives or timeouts a connection can stay in this state indefinitely.

Keep-alives could avoid this. In the meantime, you could have the client periodically send application-level pings/keep-alives or somethings similar.

[JuliusSweetland]

Thanks @halter73 so I believe, between the discussed use cases and the fact that it is possible for the recipient of a stream (client or server) to be waiting for a message which isn't coming, that gRPC Core keep alive functionality is required on both the client and server side.

@JamesNK do you agree?

@halter73 I am using manual pings and monitoring in both directions for my bi directional streaming use case, but it would be great to simplify my code and push this down to a lower level.

[JamesNK]

Keep alive pings are supported in 5.0. This can be closed.

Best practice documentation related to performance: https://docs.microsoft.com/en-us/aspnet/core/grpc/performance?view=aspnetcore-5.0#keep-alive-pings