Caddy: Proxy Protocol Support

[mikepulaski]

What?

I’d like to implement Proxy Protocol Support to help balance WebSocket connections served by Caddy in production. It’s supported by both NGINX (docs) and HAProxy (docs).

Why?

It allows us to know the origin of requests behind external load balancers which route traffic to a cluster of Caddy instances.

How?

The existing middleware API can’t be used to implement this feature because the PROXY packet would be sent _before_ Caddy even receives the HTTP method.

For example, a raw request might look like this:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
GET / HTTP/1.1\r\n
Host: 192.168.0.11\r\n
\r\n

As such, this is something that would have to be parsed, handled and made available at a much lower level. We can write a custom net.Listener to parse the data, then inject it in the HTTP headers in the server’s ServeHTTP method (i.e. X-Forwarded-For)

For end users, there’d be something that could be added in the Caddyfile to enable the use of this listener. For instance a proxy_header directive.

---

I’d appreciate any feedback. I’ll gladly implement support for this myself after discussion with you beautiful guys and gals :)

[mholt]

So this takes place below the HTTP layer? Caddy is an HTTP server... but this still might be possible as an add-on, even if that isn't possible until 0.9 (which I'm writing now).

Would this custom listener also serve regular HTTP requests?

[mikepulaski]

@mholt Yes, this would have to take place below the HTTP layer.

The custom listener would still serve regular HTTP requests; it would work similarly to the integration between server.Server and server.gracefulListener.

This listener would essentially read only the first line, parse it, and return a net.Conn variant to server.Server which would have the contextual information. If only the first line is read, then the remaining data within the net.Conn buffer should be the expected HTTP request.

[mholt]

Okay, so maybe it would be possible to just wrap the underlying code so that they still see the HTTP request as expected.

I'm not opposed to this. I'm just not sure how it would hook into Caddy, but now's the time to figure that out, since I'm working on the 0.9 rewrite, which tries to answer questions like this. Take a look at what I've got so far, although I need to update that projection with some recent changes. I can do that hopefully early next week.

So basically your plugin would need a way to wrap the listener of the server it's plugging into. Right?

[mikepulaski]

Yeah, my goal would be to make this completely seamless; HTTP requests should appear unmodified except for an additional/explicit tag: X-Forwarded-For.

This plugin would indeed need a way to wrap the listener of the server, but in order to relay this information via an HTTP header, I’d imagine we’d need to implicitly enable a bundled middleware component.

[mholt]

The middleware component shouldn't be a problem; that's already possible. Hmm, well, if you take a gander in that 0.9 branch inside the caddyhttp/httpserver folder, you'll find server.go and plugin.go, two relevant files for plugging into the HTTP server. If you have any proposals/suggestions how to go about that (even if you're not sure it'll work), let me know! That will help me get a better idea what is needed going forward.

[mikepulaski]

Ok, I’ll have a look early next week and see how it might work in your 0.9 branch. Thanks for the quick response!

[mholt]

@mikepulaski I'm a little ahead of schedule and just updated it with my current progress. Take a gander when you get the chance. It might be as simple as a function hook or callback to wrap a listener.

[mikepulaski]

@mholt At first pass, it seems this feature could be implemented by adding an attribute to SiteConfig, let’s say SiteConfig.EnableProxyProtocol. If enabled, a custom listener will be either returned by Server.Listen or privately wrapped inside of Server.Serve. Either would hide any unnecessary implementation details from callers, but I’m not sure what the implications are by having it returned by Listen. Are there any use cases where a listener would be used outside of Server.Serve?

The benefit of having it as an attribute within httpserver.SiteConfig is that we can implicitly prepend a middleware component within httpserver.NewServer to handle the parsed PROXY line so we can inject X-Forwarded-For headers.

[mholt]

@mikepulaski Thanks for the thoughts!

Are there any use cases where a listener would be used outside of Server.Serve?

As of right now, no, however I am not sure I would guarantee that in the future.

it seems this feature could be implemented by adding an attribute to SiteConfig, let’s say SiteConfig.EnableProxyProtocol. If enabled, a custom listener will be either returned by Server.Listen or privately wrapped inside of Server.Serve.

If you would like, you are welcome to draft up a pull request with what the change would look like. I can't guarantee it'll be merged right away, but if it doesn't take you too long I think it could definitely be worth a review.

One tricky thing might be in the MakeServers() function where site configs are grouped by their listeners. Sites that share a listener are of course served with the same listener. This change will muddle slightly what it means to "share" a listener (for example, TLS and plaintext sites can't share a listener because the protocols are different). Whenever possible, I like the state that's contained within a SiteConfig to belong to a site, not the whole listener, because as you will see, it makes things harder.

(Also take note of occurrences of "X-Forwarded-For" in the caddy source code, make sure it wouldn't break.)

The benefit of having it as an attribute within httpserver.SiteConfig is that we can implicitly prepend a middleware component within httpserver.NewServer to handle the parsed PROXY line so we can inject X-Forwarded-For headers.

I'm a little nervous about melding listeners and middlewares as I see those on separate levels, separated by yet another which is the HTTP server itself (and its root ServeHTTP method). But that's why a PR might help, it will help me see what you are thinking of. :)

[git001]

Now 0.9 is out ;-).
Congratulation very good job.

Where should we consider to add such a protocol in the new 0.9 infrastructure?

[mholt]

@git001 Start by implementing a listener that wraps an underlying net.Listener. Then we'll figure out a good way to wire it up into Caddy once you have a simple http.Serve() example working with it.

[samuel]

Here's an implementation of the v1 protocol (as supported by AWS ELB) that we're currently using in production: https://github.com/SpruceHealth/go-proxy-protocol

To use it with the http server need to create a custom listener and then wrap it and hand it to http.(*Server).Serve(&proxyproto.Listener{Listener: listener})

[mwpastore]

I've been using this in production: https://github.com/armon/go-proxyproto

[scalp42]

Curious if it's planned or not as it's been a while since activity on this @mholt

Links that could help:

• http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html
• http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Thanks for Caddy!

[mholt]

@scalp42 It's not on _my_ TODO list, but somebody is welcome to implement it, as long as it overcomes the technical/perf issues discussed above. And you're welcome :)

[mastercactapus]

I may be able to take a shot at this in the coming week or two.

@mholt do you know that it would cause an issue for the wrapped listener to return a wrapped Conn struct? If not, the RemoteAddr method could just return the source address instead of the underlying address.

Something like this:

type proxyConn struct {
    net.Conn
    remote net.Addr
}
func (p *proxyConn) RemoteAddr() net.Addr {
    return p.remote
}

Thus keeping it separate from any middleware/header stuff if possible -- I think this could actually be fairly clean.

[mholt]

@mastercactapus I haven't looked into that possibility much, and I haven't looked at the base server code very much in a while, either. You can definitely try that approach, I like cleanness. :grin: I'll recommend that you take care to implement any interfaces that the type you're wrapping implements, so that we don't break anything...

[mastercactapus]

Just an update: I threw something together today. I want to test it a bit more before making a PR, though.

There are 2 new flags: -proxy-protocol-subnets and -proxy-protocol-timeout

I would have added directives, but alas, 1 vhost != 1 net.Listener. So I followed what QUIC and HTTP2 were doing and added flags (since those two also happen before we know what virtual server is being requested)

Also, I implemented subnet filters instead of a boolean, to limit who could send the PROXY.... header. It's really only for traffic from the ELB or what-have-you. (one could set it to 0.0.0.0/0,::/0 if they wanted, however)

@mholt let me know what your thoughts are if you get the chance. I know adding more flags isn't ideal, but I couldn't think of a more appropriate place for what needs to happen. And I called the flags generically what they were, but the names kind of stick out -- maybe there's a better name for them that matches the other flags in caddy better.

@mwpastore @samuel I chose the https://github.com/armon/go-proxyproto package, as it made it easy to make the subnet-filtering-listener.

Last but not least, it falls back cleanly if the PROXY header is missing (but not invalid). I'm not sure if nginx and/or haproxy do this, or if they are strict and require the header. It was pretty useful in testing to not have things fail without it though. In either case, if we want it to be strict, the github.com/armon/go-proxyproto package will have to be updated.

mastercactapus/caddy@b022e2cbed5ef33b8ab3898c2cbfaf2da6b053b7
https://github.com/mastercactapus/caddy/tree/proxy_protocol

[mwpastore]

This looks great! Thanks @mastercactapus. The more difficult part is likely to be implementing PROXY protocol upstream, as it doesn't appear as though there is any existing prior art. Not in Go at least.

[mholt]

Since Caddy now supports listener middleware, Proxy protocol can be implemented as a plugin. I think we're done here!