Caddy: proxying http2 explicitly

[groob]

1. What version of Caddy are you using (caddy -version)?

Caddy 0.10.4

2. What are you trying to do?

Proxy a grpc client connection to a grpc server without using the grpc package.
Because grpc is http2 this should "just work". However, no matter what I try, http.Transport always tries to use an http1 RoundTripper, and causes a malformed response error. http2.ConfigureTransport doesn't seem to have an effect, but using http2.Transport with a custom dial method directly appears to work.

3. What is your entire Caddyfile?

https://localhost:8080 {
    tls cert.pem key.pem
    log stdout
    proxy / https://localhost:8081 {
    insecure_skip_verify

    }
}

localhost 8080 is the proxy (caddy in this case)
localhost 8081 is running the https://github.com/grpc/grpc-go/tree/master/examples/helloworld modified slightly to use a tls.Listener instead of just straight up tcp.
the client is the greeter client also using certs:

    creds, err := credentials.NewClientTLSFromFile("cert.pem", "localhost:8080")
    conn, err := grpc.Dial(address, grpc.WithTransportCredentials(creds))

4. How did you run Caddy (give the full command and describe the execution environment)?

~/Downloads/caddy_v0.10.4_darwin_amd64/caddy

5. Please paste any relevant HTTP request(s) here.

./greeter-client

6. What did you expect to see?

Proxied traffic to the greeter server.

7. What did you see instead (give full error messages and/or log)?

Activating privacy features... done.
https://localhost:8080
WARNING: File descriptor limit 7168 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192".
::1 - - [18/Jul/2017:11:03:11 -0400] "POST /helloworld.Greeter/SayHello HTTP/2.0" 502 16

The error here is a little bit cryptic, but I've seen the same 502 response from an httputil.ReverseProxy. There the error is

2017/07/18 11:08:39 http: proxy error: malformed HTTP response "\x00\x00\x00\x04\x00\x00\x00\x00\x00"

It comes from the http.ReadResponse method, which is trying to read an HTTP 1.0 response.
I tried doing various things, but always get the same error.

Workaround:

I was able to work around this issue by explicitly importing http2.Transport.
This sample reverse proxy is able to proxy grpc traffic.
https://gist.github.com/groob/66cf93cf4c799bae52ddc86c4238ad1f

Note that there's a related issue https://github.com/golang/go/issues/20437 which was fixed in Go 1.9 that addresses a problem with the headers but the malformed HTTP response error documented here is independent.

[mholt]

Thanks @groob for posting this. I hope to revisit this after Go 1.9 is out, or if someone else beats me to it, that'd be great. I'm kind of surprised manually specifying an http2.Transport is needed for this post-Go1.6...

[pieterlouw]

Hi @groob,

When you say you are trying to proxy a client without the grpc package,do you mean the Go grpc package or the grpc plugin for Caddy?

Pieter

[bcreane]

@mholt - was there any change in status for linking against http2.Transport? I'm interested in using Caddy as a reverse proxy server on HTTP/2 (as well as serving static files).

[mholt]

I'm stumped because this should work. There's also a grpc plugin you could try, for improved support.

[mholt]

Would anyone here be interested in testing Caddy 2 (currently in beta) against this? Lots of improvements have been made, so if it wasn't possible before, it certainly is possible to add this feature now. See the v2 branch.

[pieterlouw]

Hi @mholt ,
Sure, will make time for this at some point.

[freman]

Interestingly I was able to get this working in Caddy just fine (if awfully slow)
In Caddy2 it writes one request to my GRPC stream and stalls.

edit: Caddy2 config in case I'm dumb (I couldn't get self_signed certs to work so I just generated one)

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "listen": [":443"],
                    "routes": [{
                        "handle": [{
                            "handler": "reverse_proxy",
                            "upstreams": [{
                                "dial": "localhost:8372"
                            }],
                            "transport": {
                                "protocol":"http",
                                "tls": {
                                    "insecure_skip_verify":true
                                },
                                "keep_alive": {
                                    "enabled": true
                                }
                            }
                        }]
                    }]
                }
            }
        },
        "tls": {
            "certificates": {
                "load_files": [{
                    "certificate": "cert.crt",
                    "key": "cert.key",
                    "Tags": ["localhost"]
                }]
            }
        }
    }
}

I can put together a whole sample tree of project and config if that helps?

[mholt]

@freman Thanks for trying it out! That's very interesting. I wonder why it would stall. Can you provide more details what you mean by that?

I can put together a whole sample tree of project and config if that helps?

Yes, please! The easier this is for developers to reproduce, the better. (I personally have to move between a lot of issues, so doing setup before even getting to observe the behavior is a huge time sink, thus, anything you can do to make the issue as simple as possible to observe is suuuper helpful!)

(v2 has "self-signed" certs on the roadmap, maybe for 2.1 or something. I'm in early design discussions currently.)

[mholt]

@groob @freman I think we got this working after:

• https://github.com/caddyserver/caddy/issues/3236
• https://github.com/caddyserver/caddy/issues/3227
• https://github.com/caddyserver/caddy/issues/3218

in Caddy 2. The latest on master (newer than v2.0.0) will have the best support for it. Please give it a try! At this point, please open a new issue if it still does not work.