Caddy: Add h2c support for an upstream

[Zetanova]

The normal case is always to encrypt http2 network connections,
but since there are cases in gRPC and service-meshes where the upstream
is over an unix domain stocket (UDS), http2 cleartext (h2c) would be required.

There is an easy why to enable the golang client to connect over h2c
https://github.com/thrawn01/h2c-golang-example

the only requirement is to add an optional switch to the caddy-config
maybe to use the undocumented config value reverse_proxy.transport.http.versions
https://github.com/caddyserver/caddy/blob/1456f15f9a3981c5b0ca8927f14bccb2c9c12660/modules/caddyhttp/reverseproxy/httptransport.go#L56

then the reverseporxy/httptransport newTransport() function need to be update:
https://github.com/caddyserver/caddy/blob/1456f15f9a3981c5b0ca8927f14bccb2c9c12660/modules/caddyhttp/reverseproxy/httptransport.go#L92-L114

the AllowHTTP property need to be set to true if h2c is explicitly allowed (maybe set in the versions setting)

Simple http2 h2c client:

client := http.Client{
    Transport: &http2.Transport{
        // So http2.Transport doesn't complain the URL scheme isn't 'https'
        AllowHTTP: true,
        // Pretend we are dialing a TLS endpoint. (Note, we ignore the passed tls.Config)
        DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, error) {
            return net.Dial(network, addr)
        },
    },
}

I have no exp. in golang and cant make a PR for it.

[mholt]

That's very interesting! Thanks for the well-researched feature request.

I think this should be quite possible, although I'm not 100% sure about the specific implementation details yet either. Will look at it sometime (or anyone is welcome to take a stab at it and we can discuss in a PR).

[Zetanova]

The newTransport function need to be extened that it produces different kinds of http-clients.
There are 7 kinds of http-client setups and only the first 3 are currently covered.

http-client types:
1) http1
2) http1+TLS
3) http1+TLS and h2 (http2+TLS) (negate over tls-ALPN)
4) http1+TLS and h2 (http2+TLS) (negate over http-upgrade)
5) http1 and h2c (http2 cleartext) (negate over http-upgrade)
6) h2 (http2+TLS)
7) h2c (http2 cleartext)

With the help of a branch (if-else) the Type 6. and 7. can be easily be implement
to just return a http2 only client in h2 or h2c flavor depending on the config or upstream-url

Example1 Caddyfile for only/direct h2c (http2 cleartext)
with a "allow_insecure" property proposal and the versions property:

reverse_proxy {
    to unix//tmp/upstream.sock
    #to http://127.0.0.1:5216

    transport http {
        allow_insecure
        versions 2
    }
}

Example2 for only/direct h2 (http2+tls)
with a "allow_insecure" property proposal and the versions property

reverse_proxy {
    #to unix//tmp/upstream.sock
    to https://127.0.0.1:5216

    transport http {
        versions 2
    }
}

If the ´reverse_proxy.transport.versions´ config field is not used anywhere,
it could be change to:

http-client types:
1) http1 => versions: ["1"]
2) http1+TLS => versions: ["1"]
3) http1+TLS and h2 (http2+TLS) (negate over tls-ALPN) => versions: ["1", "h2"]
4) http1+TLS and h2 (http2+TLS) (negate over http-upgrade) => versions: ["1", "h2"] + ???
5) http1 and h2c (http2 cleartext) (negate over http-upgrade) => versions: ["1", "h2c"] + ???
6) h2 (http2+TLS) => versions: ["h2"]
7) h2c (http2 cleartext) => versions: ["h2c"]

I would be grateful, if Point 6 and 7 can be implement quick
Because of the branching in the newTransport function,
it will not break any running config.

[mholt]

Nice breakdown, although... I'm pretty sure Caddy 2's HTTP transport for the proxy already supports 1, 2, 3, and 6 ... and maybe 4, I'm not sure what "negate over http-upgrade" means.

https://github.com/caddyserver/caddy/blob/3d6fc1e1b7f41ebbd0c7cb48280ce1a14a56f4d5/modules/caddyhttp/reverseproxy/httptransport.go#L140-L144

The only thing from your list that it doesn't support yet is H2C.

[mholt]

@Zetanova I have made an initial attempt at implementing the H2C client on the h2c branch here: https://github.com/caddyserver/caddy/tree/h2c

Can you please build that branch and try it out? I have not tested it.

You can specify versions h2c in your Caddyfile under the transport http block. Or versions 2 h2c if you want both.

(Not really loving the inconsistency in the version names there, I might change h2c to just 2c or something.)

[francislavoie]

How about h1, h2, h2c for consistency?

[Zetanova]

@mholt Ok, thanks will test it out.

http1 + h2 is not the same as poor h2 or poor h2c

If the client/server wants to use http1+h2 then the client need to negate it
For h2c: with "Upgrade: h2c" like with websockets to h2 transport
For h2: with the TLS handshake over "ALPN"
see: https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header

I make a mistake the point 4) "http1+TLS and h2 (http2+TLS) (negate over http-upgrade)"
does maybe not exist.

gRPC transport require to use h2 or h2c because most servers force h2 or h2c on the port
and do not enable http1 (including negate) on the gRPC-Port.

h2 and h2c are the protocol names and to use 2 and 2c would be an made up name.

[mholt]

The thing is, the Versions field isn't for protocol names -- the protocol is HTTP, that's the whole point of the HTTPTransport -- it's for versions of that protocol. So "h" seems a bit redundant, and frankly "http1" and "h2" are equally, if not more, inconsistent.

@Zetanova Did that branch work for you?

[Zetanova]

@mholt I am getting a PROTOCOL_ERROR on the upstream now
"The request :path is invalid: '*'"

Caddylog:

{"level":"error","ts":1586009977.3215234,"logger":"http.log.error","msg":"stream error: stream ID 3; PROTOCOL_ERROR","request":{"method":"PRI","uri":"*","proto":"HTTP/2.0","remote_addr":"@","host":"","headers":{}},"status":502,"err_id":"jpwqhe4pn","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}
{"level":"error","ts":1586009977.3215783,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"PRI","uri":"*","proto":"HTTP/2.0","remote_addr":"@","host":"","headers":{}},"common_log":"@ - - [04/Apr/2020:14:19:37 +0000] \"PRI * HTTP/2.0\" 502 0","latency":0.0569526,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

Upstream-Kestrel:

dbug: Microsoft.AspNetCore.Server.Kestrel[39]
      Connection id "0HLUOKCQC8QB5" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel[1]
      Connection id "0HLUOKCQC8QB5" started.
dbug: Microsoft.AspNetCore.Server.Kestrel[35]
      Trace id "0HLUOKCQC8QB5:00000003": HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :path is invalid: '*'
dbug: Microsoft.AspNetCore.Server.Kestrel[36]
      Connection id "0HLUOKCQC8QB4" is closing.
dbug: Microsoft.AspNetCore.Server.Kestrel[36]
      Connection id "0HLUOKCQC8QB4" is closed. The last processed stream ID was 3.
dbug: Microsoft.AspNetCore.Server.Kestrel[2]
      Connection id "0HLUOKCQC8QB4" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HLUOKCQC8QB4" sending FIN because: "The Socket transport's send loop completed gracefully."

Still analyzing, will try to increase the log level on the upstream side.

[mholt]

@Zetanova What's your config? (Please post the full config you're using without any modifications.)

[Zetanova]

@mholt

Config that producces the Protocol_error:

http://

log {
    level DEBUG
}

bind unix//csi/csi.sock

reverse_proxy {
    #to unix//var/lib/hyperv/hyperv.sock
    to http://127.0.0.1:5216
    #header_up :authority localhost
    header_up Host localhost    
    transport http {
        versions h2c 2
    }
    #transport grpc
}

Config2 that is not working:

http://

log {
    level DEBUG
}

bind unix//csi/csi.sock

reverse_proxy {
    to unix//var/lib/hyperv/hyperv.sock
    #to http://127.0.0.1:5216
    #header_up :authority localhost
    header_up Host localhost    
    transport http {
        versions h2c 2
    }
    #transport grpc
}

upstream to an UDS with h2c is not working with error,
but that is for me currently not required

{"level":"error","ts":1586011546.9913356,"logger":"http.log.error","msg":"dial tcp: lookup /var/lib/hyperv/hyperv.sock: no such host","request":{"method":"PRI","uri":"*","proto":"HTTP/2.0","remote_addr":"@","host":"","headers":{}},"status":502,"err_id":"mbqma65cj","err_trace":"reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:362)"}   
{"level":"error","ts":1586011546.9914055,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"PRI","uri":"*","proto":"HTTP/2.0","remote_addr":"@","host":"","headers":{}},"common_log":"@ - - [04/Apr/2020:14:45:46 +0000] \"PRI * HTTP/2.0\" 502 0","latency":0.00017,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

[mholt]

@Zetanova Thanks. I'm unable to reproduce that error, though. What request are you making exactly? (Give me a curl command I can use.)

[Zetanova]

@mholt its the first gRPC client request for the handshake i think.
kubelet of kubernetes makes it.

My gRPC server is working with a regular gRPC client over http fine,
but when the gRPC-client is connecting over an UDS path directly
the kestrel gRPC server is throwing because of an hardcoded URL-validation
of the ":authority" header.

Thats why i would need to use caddy to overwrite the header to localhost it.
apache and nginx/openresty are all failing completely in this sector.

[mholt]

@Zetanova Great, we're making progress. Any chance you could get a dump of that request from your client, then, so that I can reproduce it?

My gRPC server is working with a regular gRPC client over http fine,
but when the gRPC-client is connecting over an UDS path directly
the kestrel gRPC server is throwing because of an hardcoded URL-validation
of the ":authority" header.

Can you be more specific? Does the commit I pushed not work with _any_ h2c for you? What about Caddy is broken here, exactly?

Like I said, Go does this through the Host header, converting it automatically: https://golang.org/pkg/net/http/#Request.Host (see godoc for that field).

I really just need the full original request that your client is making.

[mholt]

It's working for me with grpcurl:

$ grpcurl -plaintext localhost:80 my.custom.server.Service/Method
...
(caddy output)
2020/04/04 15:25:00.338 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "127.0.0.1:1010", "request": \
{"method": "PRI", "uri": "*", "proto": "HTTP/2.0", "remote_addr": "[::1]:58827", "host": "localhost", "headers": {"User-Agent": [""], "X-Forwarded-For": ["::1"]}}, \
"headers": {"Content-Type": ["text/plain; charset=utf-8"], "Content-Length": ["20"], "Date": ["Sat, 04 Apr 2020 15:25:00 GMT"]}, \
"duration": 0.000761162, "status": 200}

[Zetanova]

@mholt The first message a HTTP/2 client should send is this PRI message. From the
https://httpwg.org/specs/rfc7540.html#ConnectionHeader

To reproduce the header curl should be able too with
curl --http2-prior-knowledge http://localhost:80

I will write again after some tests on my side

[mholt]

Okay, thanks. To clarify, Caddy's not acting as an H2C server, only client to the upstream. You'll still need to use HTTPS for HTTP/2 to Caddy.

[Zetanova]

That can be the problem, kubelet connects to the UDS with http2-prior-knowledge
and it sends the PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n header instantly.

It seams that caddy is interpreting it as a regular HTTP1.1 request and relay this request
wrapped over HTTP/2 to the upstream server.

I have no control of the kubernetes kubelet bin,
it only connects to a UDS on the k8s-host-node with h2c (prior-knowledge).

I will test now the upstream to kestrel-grpc locally,
that we can finalize the upstream h2/h2c feature.

[Zetanova]

It seams somehow to work, but with errors in the gRPC response.
with error: Status(StatusCode=Cancelled, Detail="No grpc-status found on response.")

Request over a web-browser are working fine

test setup
[dotnet gRPC client] <-h2-> [caddy] <-h2c-> [dontet gRPC server]

dotnet gRPC client and server:

Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request starting HTTP/2 POST http://localhost/csi.v1.Identity/GetPluginInfo application/grpc 
Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executing endpoint 'gRPC - /csi.v1.Identity/GetPluginInfo'
HypervCsiDriver.HypervCsiIdentity: Debug: get plugin info request
Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executed endpoint 'gRPC - /csi.v1.Identity/GetPluginInfo'
Microsoft.AspNetCore.Hosting.Diagnostics: Information: Request finished in 13.6036ms 200 application/grpc
Exception thrown: 'Grpc.Core.RpcException' in System.Private.CoreLib.dll
An exception of type 'Grpc.Core.RpcException' occurred in System.Private.CoreLib.dll but was not handled in user code
Status(StatusCode=Cancelled, Detail="No grpc-status found on response.")

Caddylog:

{"level":"info","ts":1586017823.0511506,"logger":"http.log.access.log0","msg":"handled request","request":{"method":"POST","uri":"/csi.v1.Identity/GetP
luginInfo","proto":"HTTP/2.0","remote_addr":"172.17.0.1:36450","host":"127.0.0.1:8716","headers":{"User-Agent":["grpc-dotnet/2.26.0.0"],"Te":["trailers
"],"Grpc-Accept-Encoding":["identity,gzip"],"Content-Type":["application/grpc"]},"tls":{"resumed":false,"version":771,"ciphersuite":49196,"proto":"h2",
"proto_mutual":true,"server_name":""}},"common_log":"172.17.0.1 - - [04/Apr/2020:16:30:23 +0000] \"POST /csi.v1.Identity/GetPluginInfo HTTP/2.0\" 200 9
0","latency":0.0287046,"size":90,"status":200,"resp_headers":{"Server":["Caddy","Kestrel"],"Date":["Sat, 04 Apr 2020 16:30:22 GMT"],"Content-Type":["ap
plication/grpc"],"Grpc-Encoding":["identity"],"Trailer:Grpc-Status":["0"]}}

dotnet gRPC Server log

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST http://localhost/csi.v1.Identity/GetPluginInfo application/grpc
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'gRPC - /csi.v1.Identity/GetPluginInfo'
dbug: HypervCsiDriver.HypervCsiIdentity[0]
      get plugin info request
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'gRPC - /csi.v1.Identity/GetPluginInfo'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished in 13.6036ms 200 application/grpc

Caddyfile

https://

log {
    level DEBUG
}

tls internal {
    on_demand
}

reverse_proxy {
    to http://host.docker.internal:5216
    header_up Host localhost    
    transport http {
        versions h2c 2
    }
}

[mholt]

Thanks for your help with this. Looking forward to shipping h2c + grpc at some point!

[mholt]

To anyone reading this, I'm not familiar with the gRPC protocol enough to debug this further, I'll need some help.

[Zetanova]

@mholt the h2c upstream over tcp seams to work.
If someone ask/wants UDS support could be done over versions ["h2c_unix"]
i don't necessary need it.

the gRPC response error has nothing to do with h2c, it produces the same error with
[dotnet gRPC client] [caddy] [dotnet gRPC server]

I will try to find a public gRPC test server and open a new BUG/Issue for it.