Fabio: TCP proxying support

[jefferai]

This looks super cool. Wanted to send along a feature request that I think will make it even more useful.

At a previous job, we were using haproxy due to its ability to do not only straight TCP proxying, but straight TCP proxying while honoring the SNI header. This allows the TLS connection to be direct between a client and backend server, rather than having a proxy in the middle decrypt traffic and re-encrypt traffic to the backend. This direct connection is needed to avoid transitive trust issues, and can be especially important if you are required to adhere to various industry conformance standards like PCI.

So my feature request is: please allow straight TCP proxying while honoring the SNI header for routing.

[magiconair]

Transparent TCP proxy support was on my radar at some point but so far I didn't have the use case for it. I want to enhance a couple of other things first so this is not high on my prio list ATM.

Leaving this open as enhancement

[jefferai]

Sounds good. Just remember, when you get around to it, that SNI is a must as it's really the only way to route when doing TCP routing (since mapping ports is just awful)!

[ghost]

+1 Transparent TCP is currently a gaping hole that no one has filled.

edit: To clarify, a transparent TCP proxy that can work with a Consul/Etcd/Mesos/etc natively that can do connection draining.

[jefferai]

@rvm2015 I assume you mean, "within Fabio"? Other services are capable of doing transparent TCP proxying,and have for a long time.

[magiconair]

@rvm2015 What is your use case?

[ghost]

If Fabio had the ability to do TCP as well as HTTP I'd have a reason to put in the work to extend the backend support to include Mesos/Maratho. I'd then be able to replace both Traefik and HAProxy/Mesos-DNS with a single service and greatly simplify my stack.

[magiconair]

@rvm2015 Fabio is an FE-BE router which routes incoming traffic to different BE services based on information in the protocol (i.e. host and/or path from the HTTP request). I get the SNI use case for SSL traffic since I can get the host information and use that for routing information but how would I route an arbitrary incoming TCP stream to a BE service that can handle it? Based on ports? Which protocols are you using where you need raw TCP traffic?

[magiconair]

I think I have a couple of use cases for TCP routing now and the proxy itself is pretty simple. I just need to update the config language and dynamic listener support. First version may not support SNI though but I'm still looking into this.

[doublerebel]

:+1:

[yingfeng]

Transparent TCP proxy is useful when deploying such services as IM server, RPC services within docker container cloud, given this feature enabled, fabio could be a replacement for HAProxy totally for cloud environment.

[JonathanBennett]

Don't want to be one of those "me too" people but did anyone manage to get anywhere with transparent TCP proxying (with or without SNI)? @magiconair I know above you said you were looking at this?

Is there any WIP we can possibly help commit to if not finished? Else I might take a stab at this, as this is currently the one thing we're missing as we have lots of arbitrary TCP ports which would benefit hugely from fabio / only work through fabio.

Thanks in advance!

[magiconair]

@JonathanBennett The TCP proxy itself is not difficult and I think I have that ready somewhere. Main issue is how to configure the listeners and that they need to start and shutdown automatically which is different from the statically configured HTTP listeners. That also isn't hard but $dayjob and $kids take most of my energy right now.

I'm going to focus on the API for the multiple registries since this is a small change but has lots of potential. Then I'll work on the config change which will enable the strip prefix and weight parameter and also offer the tcp syntax. Then it isn't far from the dynamic listeners. I have some code lying around for that as well but it probably won't merge anymore.

If you want to help you could pick up one of these things. Let me know if you are interested and we can try to figure something out.

[mterron]

Hi, is anyone working on this one?

[magiconair]

@mterron It is on my list but I am working on other things right now. What is your use case?

[mterron]

@magiconair if fabio could do straight TCP+SNI (+PROXY proto?) load balancing I wouldn't need to deploy certs to fabio. This will solve a transitive trust issue where you have to trust fabio with private keys for a service itself doesn't provide. It is much easier to only deploy certs to the server/container/vm that need access to the plain text and keep an end-to-end encrypted connection, especially in environments where security is important.

[InAnimaTe]

To speak in parallel with some of the mentions earlier, I would find sni passthrough extremely desirable. I know a lot of people, including myself, like to keep encryption end-to-end and store their key/cert with the application itself, instead of maintaining it at the LB.

As a previous user said, most solutions out there which implement dynamic backends making them more juicy for micro-service deployment don't support this (fabio, traefik, hipache, etc), whilst the solutions which support this don't support dynamic backends (i.e. haproxy, nginx, caddy). Fabio could be one of the first All-In-One solutions that is flexible enough to do this!

[magiconair]

@InAnimaTe yes :)

[magiconair]

@InAnimaTe, @mterron, @jefferai et. al. here you go.

This patch implements TCP passthrough support for TLS connections and uses the SNI header for routing providing end-to-end encryption with dynamic routing!

This is working but needs testing. Please hit on it and find out where it breaks. I need to think a bit more about timeouts and edge cases.

Here is how you can test this:

./fabio -proxy.addr ':9999;proto=tcp+sni'

This will configure a listener on port 9999 which will capture the first package, parse the TLS ClientHello and extract the server name from it. Any error or no TLS will close the connection immediately. Traffic is then routed to the service which is registered with prefix <serverName>/. Note the trailing slash which is something I need to fix later.

The demo/server now has an option to provide a TLS certificate as follows:

./server -proto http -prefix <serverName>/ -cert cert.pem -key key.pem

You'll get some TLS handshake error lines from consul performing health checks. That's fine for now. I'll suppress them later.

Then run openssl as follows:

/usr/local/opt/openssl/bin/openssl s_client -tls1_2 -connect 127.0.0.1:9999 -servername <serverName>

and test with

GET /health HTTP/1.0

You should get

HTTP/1.0 200 OK
Date: Sun, 21 Aug 2016 09:18:49 GMT
Content-Length: 3
Content-Type: text/plain; charset=utf-8

OK
closed

Or with curl (make sure you use the homebrew version on OSX):

/usr/local/opt/curl/bin/curl -k -i https://<serverName>:5555/health
HTTP/2 200
content-type: text/plain; charset=utf-8
content-length: 3
date: Sun, 21 Aug 2016 09:22:34 GMT

OK

[InAnimaTe]

You sir are a God among Men.

I have a few services I plan to test this with sometime this week. Really glad you put forth the effort and time to get this initial version committed. Appreciate it!

[magiconair]

@InAnimaTe any update?

[mterron]

@magiconair is there a binary for this feature or do I have to compile it myself?

[magiconair]

Right now you have to compile it yourself.

[magiconair]

Rebased on master

[magiconair]

I'm considering merging this patch onto master and marking it as experimental since I still think some timeouts are missing. But that would give people a better opportunity to test this. Then I could tag a 1.3 release as well. Opinions?

[sean-]

+100

[magiconair]

Done. Merged to master and released v1.3. Documentation is here:

https://github.com/eBay/fabio/wiki/Features#tcpsni-proxy-support

Please test and let me know where it breaks.

[nugend]

Any chance of vanilla TCP support?

[magiconair]

@nugend yes, the tcp+sni proxy is essentially a vanilla TCP proxy with some goodies. So building the proxy isn't the problem. I need to add code to create listeners dynamically. It'll come but I'm working on something else right now.

[magiconair]

@nugend What's your use case?

[nugend]

Internal app using a custom connection oriented protocol. A number of client teams that don't really have the ability to be aware of our multiple instances.

[holtwilkins]

Hey @magiconair , I can't seem to get the ELB PROXY and tcp+sni pieces work together, should they, or am I missing something?

I.e. curl -k --resolve hello-world.blah.com:9999:10.2.211.134 https://hello-world.blah.com:9999/hello_world.html works from a machine that's able to see that private IP, so I've seemingly verified that tcp+sni is working correctly. But I'm trying to put fabio behind an AWS ELB, and whether I put the ELB into https 443 -> https 9999 or tcp 443 -> tcp 9999 w/PROXY support, it doesn't work. (Fabio was starting with just -proxy.addr ':9999;proto=tcp+sni' as its args here). The curl command to hit the ELB is just curl -k https://hello-world.blah.com/

In the https case, my curl just hangs, I'm not sure where it's hanging. Meanwhile fabio is spitting out 2016/10/11 03:51:57 [DEBUG] tcp+sni: server_name missing constantly, so I'm assuming something internal to the ELB is just spitting-out tons of request to fabio about something (the healthcheck is on 9998, so it shouldn't be that)?

In the tcp case, I get curl: (35) SSL received a record that exceeded the maximum permissible length. or curl: (35) Unknown SSL protocol error in connection to hello-world.blah.com:-9847. On the Fabio side, the only log I get for each of these attempts is 2016/10/11 04:01:16 [DEBUG] tcp+sni: TLS handshake failed. Any ideas?

[magiconair]

@holtwilkins I'll have a look

[magiconair]

@holtwilkins the PROXY protocol is only on the HTTP listener right now which will break the TCP+SNI listener as it will not be able to parse the ClientHello. I'll add that and put a patch on a branch. Could you test that?

[holtwilkins]

I sure can, just point me at the branch when it's ready!

[magiconair]

@holtwilkins I think that should do it. I'm working on making the TCP+SNI proxy more testable anyway and I'll add that to the test cases. However, that change isn't ready yet.

[magiconair]

@holtwilkins In case you didn't see the issue-1-add-proxyproto-to-tcp-sni branch: https://github.com/eBay/fabio/tree/issue-1-add-proxyproto-to-tcp-sni

[holtwilkins]

Thanks @magiconair , it worked for me!

[magiconair]

@holtwilkins Excellent. Then I'll merge it and make sure that I have a test for it in the next update.

[magiconair]

Merged this to master and referred to it as #177

[magiconair]

@nugend I'm tracking the generic tcp proxy support separately in #179 but I'll keep this one open since it has the comprehensive discussion of use cases in a single place.

[magiconair]

Generic TCP proxying support is now in the release-branch-1.4 Please check #179 for more info on how to use it. I'd really appreciate any feedback.

[jefferai]

I thought you weren't supposed to ever close bug #1 :-D

Great job!