Grpc-go: ServeHTTP doesn't work without TLS

[tamird]

This is somewhat obvious, knowing that Go's http2 doesn't work without TLS.

Still, it can be surprising given that grpc exposes grpc.WithInsecure(). The errors returned in this case are also really not helpful.

This is a pain point for us in CockroachDB, where we want to allow folks to operate the database without provisioning certificates.

[iamqizhao]

Non-TLS case needs some cmux type of connection dispatching solution we designed but not implemented.

But in general, if your system allows insecure connections, I do not see the reason you need to stick to a single port shared by http2 and grpc. You can have a grpc server and http server listening on 2 different ports in this case.

[tamird]

At the very least, this is a documentation issue.
On Feb 16, 2016 03:55, "Qi Zhao" [email protected] wrote:

Non-TLS case needs some cmux type of connection dispatching solution we
designed but not implemented.

But in general, if your system allows insecure connections, I do not see
the reason you need to stick to a single port shared by http2 and grpc. You
can have a grpc server and http server listening on 2 different ports in
this case.

—
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc-go/issues/555#issuecomment-184582109.

[tamird]

@iamqizhao can you reopen this please? It is actually possible to make this work with some hacks https://github.com/cockroachdb/cockroach/commit/c036c1503cdb3756da4f5c911b77253e62706a45#diff-4bf1ae5b9eb22814a15582886403053f, however Go 1.6 contains a change that rejects insecure gRPC connections before they even get to the handler https://github.com/golang/go/commit/6e11f45ebdbc7b0ee1367c80ea0a0c0ec52d6db5, which breaks this again.

cc @bradfitz

[iamqizhao]

Please provide your use case to justify why your system cannot use 2 ports in the insecure case if you think we should pursue this further. It is not that hard to implement cmux in grpc but we want to focus on the real demands given the limited resource from our side. I think insecure port sharing is a really rare case and not needed by >99% grpc users right now.

[tamird]

OK, but is closing the issue really appropriate? You don't have to fix it
right now, but it's not fair to say that the use case is illegitimate.

On Wed, Feb 17, 2016 at 8:37 PM, Qi Zhao [email protected] wrote:

Please provide your use case to justify why your system cannot use 2 ports
in the insecure case if you think we should pursue this further. It is not
that hard to implement cmux in grpc but we want to focus on the real
demands given the limited resource from our side. I think insecure port
sharing is a really rare case and not needed by >99% grpc users right now.

—
Reply to this email directly or view it on GitHub
https://github.com/grpc/grpc-go/issues/555#issuecomment-185499151.

[bradfitz]

We could use the Github Milestone feature to track prioritization. And/or Labels.

[iamqizhao]

To be clear, I was not saying the use case is illegitimate (I was actually saying we even have a designdoc for that). :)

I am trying to clean up and organize (milestones and labels, etc.) all the issues in the repo for GA. I'll see what the best way is to deal with this kind of issues. Reopen this for now.

[vgough]

I just came by to say that don't believe insecure port sharing to be rare. It is very convenient to setup grpc (& other http services) on insecure ports during development, and it is also convenient to reduce the number of ports in order to simplify firewalls and avoid port conflicts. I had been in the process of trying to consolidate ports for a project when I came across this issue.

[glerchundi]

I do not see the reason you need to stick to a single port shared by http2 and grpc. You can have a grpc server and http server listening on 2 different ports in this case.

What about grpc-gateway and grpc server working on the same endpoint? For example like @philips did in his philips/grpc-gateway-example repo.

[AlekSi]

I also do want to have gRPC and REST APIs on the same port. It is possible with TLS (see link in the previous comment), but not without it. And local developing with self-signed certificates is still cumbersome.

[AlekSi]

Cross-reference to implement a workaround with cmux:
https://open.dgraph.io/post/cmux/
https://github.com/soheilhy/cmux

[dhrp]

Sorry for the little self-promotion, but I've written a comprehensive article about this issue, in particular about the issues with ServeHTTP() for the purpose of multiplexing to the grpc-gateway. I also discuss some alternatives... I struggled with the details of this issue and think others may not need to.

Why choose between gRPC and REST?
Options for letting your service expose both REST and gRPC API’s on a single port.

[dfawley]

@peter-edge did you say you were able to accomplish this? Is there anything you could share about it?

[dfawley]

Also: https://github.com/golang/go/issues/14141

[bufdev]

I was, but it's in internal code, I have to figure out how to share it. It's not a secret, just need to go through my notes.

[snowzach]

I finally figured this one out. You can wrap your handler function in the h2c helper if you want to disable TLS

import (
         "net/http"
    "golang.org/x/net/http2"
    "golang.org/x/net/http2/h2c"
)

s.server = &http.Server{
    Addr: ":8080",
    Handler: h2c.NewHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if r.ProtoMajor == 2 && strings.Contains(r.Header.Get("Content-Type"), "application/grpc") {
            grpcServer.ServeHTTP(w, r)
        } else {
            mux.ServeHTTP(w, r)
        }
    }), &http2.Server{},
}

[nhooyr]

Given @snowzach's solution and h2c being implemented in net/http, this can be closed afaict.

[glerchundi]

In case someone is interested I came up with a solution to share ports for gRPC and a HTTP server without TLS and by avoiding the use of gRPC ServeHTTP, which is much less performant than the Serve. Basically is a combination of the lego bricks that people did in the last months: cmux + grpc.Serve + h2c:

package main

import (
    "log"
    "net"
    "fmt"
    "context"
    "net/http"

    "golang.org/x/net/http2"
    "golang.org/x/net/http2/h2c"
    "github.com/soheilhy/cmux"
    "google.golang.org/grpc"
    pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

// grpcServer is used to implement helloworld.GreeterServer.
type grpcServer struct{}

// SayHello implements helloworld.GreeterServer
func (s *grpcServer) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
    log.Printf("Received: %v", in.Name)
    return &pb.HelloReply{Message: "Hello " + in.Name}, nil
}

type httpServer struct{}

func (*httpServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello %s", "my friend")
}

func main() {
    // Create the main listener.
    l, err := net.Listen("tcp", ":23456")
    if err != nil {
        log.Fatal(err)
    }

    // Create a cmux.
    m := cmux.New(l)

    // Match connections in order:
    // First grpc, then HTTP, and otherwise Go RPC/TCP.
    grpcL := m.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))
    httpL := m.Match(cmux.HTTP1Fast())

    // Create your protocol servers.
    grpcS := grpc.NewServer()
    pb.RegisterGreeterServer(grpcS, &grpcServer{})

    httpS := &http.Server{Handler: h2c.NewHandler(&httpServer{}, &http2.Server{})}

    // Use the muxed listeners for your servers.
    go grpcS.Serve(grpcL)
    go httpS.Serve(httpL)

    // Start serving!
    m.Serve()
}

Hope you found it useful!