Dgraph: GRPC muxing causing client connection hang on setup

[markphilpot]

From the source code, it looks like the listener is muxing both http2/grpc traffic and http traffic into the same port. It looks like all grpc communication must have the content-type: application/grpc header in it.

I don't think this will always hold true, which is why I'm having issues... Using grpc-java and the Netty channel provider, the opening HTTP2 frame contains three subframes... Magic, Settings, and Window Update

It seems like the window update negotiation happens without a content-type header... and thus is not being routed by dgraph to the correct handler. Below is some output from Wireshark

I generated the java clients from the official java protobuf plugins. The java code looks like:

        ManagedChannel channel = NettyChannelBuilder
                .forAddress(
                        "docker",
                        8080
                )
                .usePlaintext(true)
                .keepAliveTime(
                        60,
                        TimeUnit.SECONDS
                )
                .build();
        DgraphGrpc.DgraphBlockingStub blockingStub = DgraphGrpc.newBlockingStub(channel);

        Check checkRequest = Check.newBuilder().build();
        Version version = blockingStub.checkVersion(checkRequest);

The java client send the first HTTP2 frame (frame 5 in the wireshark output above) and then hangs indefinitely waiting for a response.

[markphilpot]

Looking at the GRPC spec, I don't believe every HTTP2 frame will necessarily include that content-type header, so I don't think it can be used for muxing.

https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md

I would recommend just having a separate port for GRPC communication.

[manishrjain]

@srh : Have a look at the internal library that we're using, and raise this issue with them as well. We're using a lib from CockroachDB, who have forked a lib to do this muxing. So, they must have seen this issue as well.

[srh]

It looks like CockroachDB wound up going with 2 ports.
https://www.cockroachlabs.com/blog/a-tale-of-two-ports/

[srh]

Also, they don't expose gRPC to any client protocol (I think? They have a dead PR for that, and I can't find any documented gRPC interface) so they wouldn't have encountered this specific situation, just Golang<->Golang gRPC.

[markphilpot]

I would hate to give up the gRPC client interface... A separate, dedicated port for gRPC should hopefully fix the issue.

[srh]

For debugging/test scripting purposes I will also add a --port_offset flag, so that you can run machines locally with different ports just by specifying --port_offset N with different values of N. Machines will listen on ports 8080 + N, ~9080~ ~4080~ 9080 + N (for gRPC), and 12345 + N. The flag still gets considered if you specify --port, --grpc_port, or --workerport.

Edit: Going with 4080.

[srh]

gRPC will be served on port 9080 as of 302486faa4bfef316082fac3b2ca80728f3e3bc5.