Envoy: Proxy TCP connections over HTTP2 connection pools

[simonsj]

I'd like to use Envoy as something like a stunnel replacement, but leveraging long-lived HTTP2 connection pools, like this:

client <-----> E1 <-----------> E2 <-----> server
         raw       encrypted         raw
         TCP       conn. pooled      TCP
                   HTTP2 tunnel

^ where neither the client nor server speaks any HTTP.

Clients use servers' IP addresses directly, so no DNS is involved.

I was thinking it should be possible to map TCP flows like that over long-lived HTTP2 tunnels.

After reading over issues and the mailing lists and experimenting a bit locally, it's not clear to me that this kind of mode exists today as-is. But I could have missed something.

Is it possible to configure Envoy for something like this? Would it be feasible to add such a mode?

[mattklein123]

This is not supported out of the box. What does the HTTP/2 tunneling give you exactly? Are the TCP connections short lived? If this is needed a custom protocol is required. The best bet is probably some type of bi-directional gRPC streaming API that gets converted back to raw TCP on either side. You could build Envoy filters fairly easily to accomplish this.

[simonsj]

What does the HTTP/2 tunneling give you exactly?

What I'm after to is multiplex multiple streams over the single encrypted HTTP/2 connection, trying to avoid needing to set up one new connection per raw TCP flow. I'd ideally be able to reuse that connection for lots of flows over time.

Are the TCP connections short lived?

Yes, in general; at most these connections could last up to 2 hours but most would fall into the seconds range.

If this is needed a custom protocol is required. The best bet is probably some type of bi-directional gRPC streaming API that gets converted back to raw TCP on either side. You could build Envoy filters fairly easily to accomplish this.

Thanks, this is helpful, and good to know there wasn't something in-the-box that I missed. I'll study up on the filter code.

[mattklein123]

@simonsj if you potentially want to implement I can provide tips / rough overview. If this is something that the community would find useful we can upstream also.

[simonsj]

@simonsj if you potentially want to implement I can provide tips / rough overview. If this is something that the community would find useful we can upstream also.

@mattklein123 that would be awesome, I'd love to give this a try.

[mattklein123]

@simonsj here is a rough outline:

1. I would define a gRPC streaming API that basically passes a trivial message back and forth. Each message would just contain raw bytes (you might potentially want to carry some kind of routing data along with the bytes but that's separate).

2. You could follow along with tcp_proxy filter: https://github.com/envoyproxy/envoy/blob/master/source/common/filter/tcp_proxy.h. Specifically here: https://github.com/envoyproxy/envoy/blob/master/source/common/filter/tcp_proxy.cc#L207. Instead of making a raw TCP connection, you could instantiate a typed gRPC client following along with here: https://github.com/envoyproxy/envoy/blob/master/source/common/config/grpc_subscription_impl.h#L24

The above would give you the egress side of the equation. Then for ingress, you would write an HTTP filter which basically does tcp_proxy filter but in reverse.

1. Parse gRPC incoming stream using grpc codec.
2. Make TCP connection to local host or wherever.
3. Feed bytes.

[mattklein123]

Switching this to a feature request. I think Google needs this also so they might implement. cc @alyssawilk

[PiotrSikora]

Please don't reinvent the wheel. This should use extended CONNECT method proposed in https://tools.ietf.org/html/draft-mcmanus-httpbis-h2-websockets, and proxy TCP stream as a HTTP/2 stream without any extra gRPC encapsulation.

client <-----> E1 <-----------------> E2 <-----> server
         raw       HTTP/2 connection       raw
         TCP       :method = CONNECT       TCP
                   :protocol = tcp

[mattklein123]

@PiotrSikora agreed. I was not aware of this proposal. Yes, let's implement this for sure.

[songole]

@PiotrSikora @mattklein123 We are also interested in this solution. However, the rfc is a draft and I guess the preferred solution may not be available anytime soon. Do you still recommend the previous approach in the mean time? Or, is there an easier solution with Websocket and avoid HTTP/2 ? The payload to tunnel for us is HTTP/1.1.

[PiotrSikora]

@songole I don't have cycles to work on it right now, but I do recommend implementing the draft, the final version won't differ much from it (if at all).

[josdotso]

Is this kind of thing (multiplexed-TCP-proxy-over-one-upstream-listener) possible today in Envoy regardless of HTTP2? I'm trying to tunnel a bunch of TCP stuff to a single ELB port with mTLS and demux/terminate TLS on the backend.

Very loose idea of my use case:

[mattklein123]

@josdotso no, not possible today to multiplex TCP out of the box. This issue tracks developing some solution to the general problem.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[apoorvmittal10]

Is this kind of thing (multiplexed-TCP-proxy-over-one-upstream-listener) possible today in Envoy regardless of HTTP2? I'm trying to tunnel a bunch of TCP stuff to a single ELB port with mTLS and demux/terminate TLS on the backend.

Very loose idea of my use case:

@josdotso Did you achieve the mentioned architecture with dual envoy deployment and bi-directional streaming over tcp proxy? Please if you can help me with the setup and configuration too.

[alyssawilk]

I think this was closed out a bit over-enthusiastically. Envoy doesn't have the ability to mux arbitrary TCP over HTTP/H2 upgrades and then downgrade after some number of hops. I strongly suspect we'll be adding this functionality over the next year or two but it's not there today.

[alyssawilk]

Ok, I think I actually have the cycles to make this happen this/next release.

Anyone interested in bike-shedding on config with me? I've got a WIP pr for the TCP -> HTTP side of this, with a simplistic hacked config and hard-coded upgrade headers.

I assume for TCP->HTTP config we're going to want to configure tunneling protocol (today HTTP/2 only, but maybe HTTP/3 some day?). I also assume folks will want to configure both host and path which go in the Envoy-synthesized headers to be able to choose what gets mapped to which TcpUpstream on the Envoy which is doing HTTP -> TCP

For the upgrade mechanism I'd be inclined towards a custom upgrade string with Envoy in it (envoy-http2-tunneling?) which we can make configurable when/if folks need it but that may be me just reflecting mechanics I'm used to - open to other ideas!

[PiotrSikora]

This should use :protocol: bytestream when using HTTP/2 CONNECT, instead of a custom string (assuming that we don't do anything implementation specific, there).

There is also a question of how this should be exposed in Envoy:
1) if this is exposed/terminated at the HTTP layer, then it can be only matched against HTTP routing rules,
2) if this is exposed/terminated as the transport socket, then it can be matched against filter chains rules based on the stream's metadata, etc.

Also, @vasilvv was pitching tunneling {UDP,TCP}-over-HTTP/{1,2,3} at the last IETF as "WebTransport", so it might be good idea to have a chat with him.

[dio]

cc. @mdamt

[apoorvmittal10]

Anyone interested in bike-shedding on config with me? I've got a WIP pr for the TCP -> HTTP side of this, with a simplistic hacked config and hard-coded upgrade headers.

@alyssawilk I have some cycles so please let me know if I could take up something.

[alyssawilk]

As said over on 1451 Envoy connect support is ready to play with.
I've got a couple of PRs in flight improving docs and cleaning up internal code flow, but feel free to start playing around with it!