Grpc-go: rpc error: code = Unavailable desc = transport is closing error happened when launching multiple goroutines to send request by grpc client

[yangliCypressTest]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

libprotoc 3.5.1

What version of Go are you using (go version)?

go version go1.9 darwin/amd64

What operating system (Linux, Windows, …) and version?

Mac 10.11.6

What did you do?

We had one use case that: sending the requests through goroutines by grpc clients to a same endpoint simultaneously, but there is a grpc error happened "rpc error: code = Unavailable desc = transport is closing error happened", this error is not shown every time,but frequently.
And if sending requests through goroutines but to different endpoints, it can work well

What did you expect to see?

I expected that the grpc is able to support handling the client requests concurrently.

What did you see instead?

[MakMukhi]

@yangliCypressTest Thanks for reporting.
Can you clarify a few things please:

1. gRPC version. libprotoc is not gRPC version.
2. Is the endpoint a gRPC server as well?
3. Is there a proxy in between.
4. Can you turn on verbose logging and print the logs here. Set the following two env variables to turn on all logging.

GRPC_GO_LOG_SEVERITY_LEVEL=info
GRPC_GO_LOG_VERBOSITY_LEVEL=2

[yangliCypressTest]

1. How to check the grpc version (I am a newer to grpc)
2. Yes
3. No, it is happened in local env without any proxy configuration
4. I tried to configure these two variables in system env, and add grpclog.Errorf around the error, the error message is still "transport is closing", did I miss anything else?

[MakMukhi]

Did you get gRPC via go get or by vendoring a specific version?
Also, we'll need more information; a reproduction of the issue would be the most ideal case.
Meanwhile, you could also share all the logs from the client and the server.

Note that this is a use case that gRPC supports and has integration tests for it too. If there's a bug, we'll need help zeroing it down.

[dfawley]

@yangliCypressTest do you have any updates on this issue?

[yangliCypressTest]

we found the root cause yet, will close this issue. Thank you

[arindam2]

@yangliCypressTest Could you share some details of the root cause, it may help others debug similar issues.

[sam-robison-kerson]

@yangliCypressTest i am also having a similar issue please share what fixed it

[yangliCypressTest]

some implemention error, the global variables of grpc client were used by mistake.

[malengatiger]

Please explain the answer - this is not enough for us to know what to do to resolve this

[menghanl]

@malengatiger "transport is closing" error is often caused by the server closing the connection. Try to look at the server side logging and see if there's any transport error there.

[vaishali-ntnx]

Even I am facing this issue. Server side logs are as below -

INFO: 2018/10/13 23:19:32 parsed scheme: ""
INFO: 2018/10/13 23:19:32 scheme "" not registered, fallback to default scheme
INFO: 2018/10/13 23:19:32 ccResolverWrapper: sending new addresses to cc: [{localhost:10000 0  <nil>}]
INFO: 2018/10/13 23:19:32 ClientConn switching balancer to "pick_first"
INFO: 2018/10/13 23:19:32 pickfirstBalancer: HandleSubConnStateChange: 0xc000148060, CONNECTING
grpc on port: 10000
INFO: 2018/10/13 23:19:32 pickfirstBalancer: HandleSubConnStateChange: 0xc000148060, READY
INFO: 2018/10/13 23:19:32 transport: loopyWriter.run returning. connection error: desc = "transport is closing"
INFO: 2018/10/13 23:19:32 pickfirstBalancer: HandleSubConnStateChange: 0xc000148060, TRANSIENT_FAILURE

Can anyone help me in understanding this issue?

[dfawley]

@vaishalig2693 - Those logs are client-side logs, not server-side.

These errors are happening extremely early in the connection. You could try setting grpc.WithWaitForHandshake in your DialOptions and see if anything else shows up in the logs. My guess is the connection was never actually valid. Are you using TLS or WithInsecure?

[vaishali-ntnx]

I am using WithInsecure. Looks like as pointed by one of the issues, it doesn't work with "WithInsecure"

[dfawley]

@vaishalig2693 - everything should work correctly with WithInsecure if the server is configured accordingly. My guess is you are not connecting to the correct host/port, or the server is expecting your client to use TLS, not plaintext.

[vaishali-ntnx]

I had been using this example - https://github.com/philips/grpc-gateway-example/issues/22. I only changed TLS options with WithInsecure and was getting "transport is closing" again and again.
In this issue, it has been suggested to use cmux if we want to have common insecure port for REST and grpc

[angadn]

FWIW server-configuration is what solved the problem for me: https://stackoverflow.com/a/54703234/382564