Grpc-java: How do you tell the difference between various INTERNAL errors?

[DylanLange]

Context:

I'm working on an Android app which connects to a server with a GRPC stream. The server periodically will send RST_STREAM, resulting in io.grpc.stub.StreamObserver.onError being called in my stream response observer.

When calling io.grpc.Status.fromThrowable(cause) and inspecting the resulting Status object, I can see the following Status object: Status{code=INTERNAL, description=Internal error\nRst Stream, cause=null}.

We need to be able to tell apart from other INTERNAL errors, but it appears the only way to know it was caused specifically by RST_STREAM coming through is by looking at the description field provided.

Question:

Is there any way that we would be able to tell the difference between a RST_STREAM triggered INTERNAL error, and any other INTERNAL error in a way which isn't checking the description string contains "Rst Stream"?

If not, is there any chance that this sort of detail is exposed somehow to users of this library?

Thanks!

[ejona86]

It seems like your server is occasionally producing RST_STREAM(INTERNAL_ERROR). That should not be happening regularly. Do you know why that is happening?

INTERNAL should not generally be reacted to in any special way. They tend to be bug-worthy and not just "normal errors."

[DylanLange]

Yeah I agree. It's definitely not ideal on my side. Unfortunately I don't think we have control over _how exactly_ it kills the stream.

Just wanted to get back to this thread and let ya know I'm enquiring further about this on my side, and will be back to this question once I understand exactly what's going on at the server level. I was hoping there would be something simple that I missed 😅

Would you rather I closed this question for now and reopen/create a new one later on if necessary @ejona86?

[ejona86]

It's been a couple weeks now. Let's close this and if you comment back on it we can reopen it. Or you can create a new issue.

I do hope you find the source of the INTERNAL_ERROR, since that really should mean there's a bug.

[DylanLange]

So it turns out it's a load balancer which is cutting the connection. It seems that the way to detect this is to check if the gRPC status code != 200, and there are no trailers. Otherwise, we can assume it's a legitimate INTERNAL error sent from the server.

Thanks @ejona86

[ejona86]

Why did the load balancer cut the connection? Did it experience an internal crash? If not, it should not be using http/2's INTERNAL_ERROR it seems. It would be bad to retry for INTERNAL_ERROR because that can increase the impact from queries of death.

If this is a public load balancer, I'd be willing to discuss more directly with them.

[DylanLange]

Can't really go in to the details because it's not public facing, sorry. This is something I'm going to need to sort out with the team internally.

[ejona86]

I'm going to close this because there doesn't seem much we can do. The only option I see is to treat HTTP/2 INTERNAL_ERRORs as UNAVAILABLE instead of INTERNAL. While that is possible, it seems like a bad idea as it could cause retries for queries-of-death.

If there's some serious reason that the proxy used INTERNAL_ERROR yet the RPC should be retried, then that could be provided and we can reopen the issue and look into it. Honestly, INTERNAL_ERROR is one of those things that "shouldn't happen," so I doubt it makes sense to make any changes here and instead focus on the root causes. If we or another implementation has broken behavior causing INTERNAL_ERRORs, we would be _very_ interested in resolving those.