Grpc-node: Channel State Does Not Change

[raybooysen]

Problem description

If we follow these steps
1) Connect to a GRPC server.
2) Periodically (every 15 seconds), issue a request response to the server as a heartbeat.
3) Unplug the network cable
4) The GRPC client starts erroring with a TCP error
5) The Channel will stay with a state of 2 (connected)

What this means is we cannot rely on the channel to know the REAL connectivity state. I'm unsure if this is expected. Should the channel stay connected until:
1) The OS kills the socket
2) Or a TCP error happens when issuing an RPC?

Environment

• OS name, version and architecture: Linux Mint 19
• Node version [e.g. 8.10.0] 10.15.1
• Node installation method [e.g. nvm] yarn
• Package name and version [e.g. [email protected]] [email protected]

Additional context

The real question I'm trying to get to the bottom to is how, with GRPC, to know whether an endpoint is healthy or not. Currently we rely on heartbeats, ignoring the connection state completely. However, that causes a lot of noise in the logging when laptops wake up from sleep or connections are switching from wifi to network.

Thanks for a great product, hoping for some guidance.

[nicolasnoble]

This is by design, yes. Something like this is considered a transient failure, meaning that grpc should trigger its reconnection logic without bubbling it up.

[raybooysen]

Which is all fine but how can we know? We want to power a connection status
monitor. While you say this is a transient failure, the channel doesn't
move into the transient failure state. Should it?

On Wed, 20 Feb 2019, 23:31 Nicolas Noble, [e-mail] wrote:

This is by design, yes. Something like this is considered a transient
failure, meaning that grpc should trigger its reconnection logic without
bubbling it up.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-node/issues/745#issuecomment-465626610, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAcMhUDe7mfLwKDcCX218B55Jijm0wrIks5vPWpQgaJpZM4bEcg3
.

[murgatroid99]

I just want to verify: are you saying that the channel is reporting a READY connectivity state while the network cable is unplugged and the computer is completely disconnected from all networks? Also, how and when exactly are you querying the connectivity state?

In general, the gRPC library can report other connection states, including CONNECTING and TRANSIENT_FAILURE, but dependent on how you are querying them and the state of the network, it is possible for the gRPC library to disconnect and reconnect in between checks.

[nicolasnoble]

Right, I have read your question a bit too fast: it is expected that if a channel is established, and you are sending RPCs periodically through it, having the network cable disconnected is going to be transparent to the activity of sending RPCs. But the channel state itself accessible from an API (that I personally dislike, but I digress) should reflect the transient failure state properly.

[raybooysen]

We are asking for the state from the channel via the asynchronous API with
an infinite deadline. While waiting, we unplug the cable and disconnect
WiFi.

RPCs fail but the channel is still in a ready state.

On Thu, 21 Feb 2019, 06:16 Nicolas Noble, [e-mail] wrote:

Right, I have read your question a bit too fast: it is expected that if a
channel is established, and you are sending RPCs periodically through it,
having the network cable disconnected is going to be transparent to the
activity of sending RPCs. But the channel state itself accessible from
an API (that I a personally dislike, but I digress) should reflect the
transient failure state properly.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-node/issues/745#issuecomment-465794641, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAcMhQ2SBd2uzrsgBbSHzSV7B8R3Umsiks5vPddigaJpZM4bEcg3
.

[murgatroid99]

That does sound like a bug to me. Can you share the exact code you have for using that API, so that I can see how to reproduce this problem?

[raybooysen]

Sure. Will be back at work on Monday and will put together a sample.

Thanks for the quick responses

On Thu, 21 Feb 2019, 08:17 Michael Lumish, [e-mail] wrote:

That does sound like a bug to me. Can you share the exact code you have
for using that API, so that I can see how to reproduce this problem?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/grpc/grpc-node/issues/745#issuecomment-465821984, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAcMhcvY7RxCIjSc1X1YXUsohRfwWvmOks5vPfOwgaJpZM4bEcg3
.

[nicolasnoble]

It might also be useful to get a full debug log. Sometimes, kernels can be a bit whacky when dealing with events like unplugging the network cable. I wouldn't be too surprised if we did not get a signal up from the network stack in your case. I once witnessed a dead network link leaving network sockets still opened and slowly filling up with data for days before the kernel finally decided to signal the problem up.

[nicolasnoble]

To be a bit more specific, the answer to your question

Should the channel stay connected until:

1. The OS kills the socket
2. Or a TCP error happens when issuing an RPC?

is "yes". We do not interface with the kernel beyond the socket. If the OS doesn't actually kill it, then, no, we wouldn't be able to report any change of state.

[raybooysen]

I was wrong on my assertions about the issuance of the RPC. When the RPC is hit, I get DEADLINE_EXCEEDED, not a TCP error. this was my mistake.

Does this mean the only way to accurately detect a broken connection is through manual heartbeat failure detection?

[nicolasnoble]

Not necessarily. The http2 protocol includes a keep alive mechanism which should exercise the tcp layer enough so that the channel state API should reflect what's going on. But this is still going to fully rely on the status of the TCP connection reported by the kernel.

[raybooysen]

OK, if not necessarily, what is the optimal way? We have a requirement to show a dashboard of channels, at the moment with the cable unplugged we have to wait until the OS to let us know it's down, unless we rely on a combination of the channel state and heartbeating.

Is that a sensible solution?

[murgatroid99]

The http2 keepalive mechanism in gRPC actually has some extra functionality that can help here. It does in fact detect if the pings it uses are not acknowledged and considers the connection dropped. However, doing this in a way that detects dropped connections quickly will add a lot of steady-state traffic to all of your connections.

The options that can be passed when constructing clients or channels, including keepalive options, are defined here. You would definitely want to set "grpc.keepalive_time_ms" and "grpc.keepalive_timeout_ms". If you have long-running calls you will also want to set "grpc.keepalive_permit_without_calls". If your channel generally spends long periods of time idle you may also need to modify the "grpc.http2.max_pings_without_data" and "grpc.http2.min_time_between_pings_ms" arguments.

[raybooysen]

Thanks @murgatroid99