Grpc-node: Port not included in authority, leading to routing failures

[seeruk]

Problem description

Essentially, this looks the same as this issue: https://github.com/grpc/grpc-go/issues/1120

In this case, @grpc/grpc-js doesn't include the port in the authority when making requests. If you're in an environment that uses the authority for routing (for example, Linkerd2), then routing will fail as the sidecar will proxy to port 80 (well, will fail unless your gRPC server is using port 80).

The grpc node package doesn't suffer from this issue, and does include the port number, I've currently reverted to using that (but nice work on making that a very easy transition!)

Reproduction steps

• Use @grpc/grpc-js to make a request to a gRPC service.
• Observe the authority doesn't include the port.

You could also specifically try to deploy a @grpc/grpc-js application into Kubernetes with Linkerd2, then try to make a request to a gRPC service from a @grpc/grpc-js client, it should fail.

For comparison, here's what $ linkerd tap looks like for the gRPC client application:

With grpc:

req id=0:2 proxy=out src=172.30.0.16:48814 dst=172.30.0.18:50051 tls=true :method=POST :authority=client.default.svc.cluster.local:50051 :path=/namespace.Service/Method
rsp id=0:2 proxy=out src=172.30.0.16:48814 dst=172.30.0.18:50051 tls=true :status=200 latency=22078µs
end id=0:2 proxy=out src=172.30.0.16:48814 dst=172.30.0.18:50051 tls=true grpc-status=OK duration=76µs response-length=89B

With @grpc/grpc-js:

req id=0:1 proxy=out src=172.30.7.39:49006 dst=172.30.1.10:80 tls=true :method=POST :authority=client.default.svc.cluster.local :path=/namespace.Service/Method
rsp id=0:1 proxy=out src=172.30.7.39:49006 dst=172.30.1.10:80 tls=true :status=200 latency=37035µs
end id=0:1 proxy=out src=172.30.7.39:49006 dst=172.30.1.10:80 tls=true grpc-status=Unavailable duration=28µs response-length=0B

Environment

• OS name, version and architecture: Linux x64 (Google, GKE, 1.16)
• Node version: v12.18.0
• Node installation method: Docker image (erbium-alpine)
• Package name and version: @grpc/grpc-js v1.0.5

[murgatroid99]

Unfortunately, I had to remove the port from the authority to fix #1347. I never entirely understood what caused that bug, so it will take some work to figure out how to fix this issue without regressing on that one.

[seeruk]

Aah, that's quite an odd looking one indeed... For now at least grpc is still there. Strange that other gRPC implementations that do include the port in the authority don't suffer from this issue... Or do they? Was that other issue tested with grpc, given grpc does send the port in the authority?

The only other thing I can think of is that maybe this is an issue with the other Google API libraries, and not this gRPC one? It may have just surfaced a bug in how those libraries (or services behind them?) are handling gRPC requests?

[murgatroid99]

As far as I know, PubSub, the library referenced in the issue I linked, actually works just fine if you substitute grpc for @grpc/grpc-js. So, that shouldn't be the issue, but it's pretty clear that adding the port caused failures, and removing the port fixed the failures. So, I don't feel comfortable putting the port back and assuming the problem is elsewhere without understanding what's going wrong in the first place.

[murgatroid99]

I think I have found the problem. grpc-js uses the same string for the :authority header and the host part of the service_url passed to credentials callbacks. grpc keeps the port in the former, and omits it from the latter. The issue I linked as about a credentials error, so I think that was the problem.

[seeruk]

Aha, fantastic! Thanks for looking into that so so quickly. I look forward to seeing this released.

[murgatroid99]

That fix is out in version 1.1.0.

[seeruk]

I can confirm it's fixed the issue I was seeing with Linkerd - thanks very much!