Contour: SSL Passthrough

[rusenask]

Hello, it would be good to know whether Contour supports ssl passthrough and if it doesn't - whether it would be possible/reasonable to add it.

NGINX ingress controller has this ingress.kubernetes.io/ssl-passthrough: "true" annotation, it might make sense to keep the format the same to make it easier to switch between ingresses.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: contour
    ingress.kubernetes.io/ssl-passthrough: "true"
  name: relay-ingress
  namespace: default

[rothgar]

related to #14

[davecheney]

@rusenask Thanks for raising this issue. I need to ask a few clarifying questions to check I understand what a successful outcome of this issue would look like.

q. In this scenario who is doing TLS processing? Is it the load balancer in front of Contour? Is it Contour? Is it the service the ingress points to? Is it two or maybe all three?

q. If TLS processing is being done by the final k8s service, it feels like you'd want to configure Envoy as a TCP proxy, not a HTTPS proxy. Did I get that right?

[rusenask]

@rothgar that issue should probably be configured through another annotation:
ingress.kubernetes.io/secure-backend: "false/true"

@davecheney

q. In this scenario who is doing TLS processing? Is it the load balancer in front of Contour? Is it Contour? Is it the service the ingress points to? Is it two or maybe all three?

Service that ingress points to. LB and Contour would just pass through traffic to the destination, no termination should happen.

q. If TLS processing is being done by the final k8s service, it feels like you'd want to configure Envoy as a TCP proxy, not a HTTPS proxy. Did I get that right?

Not sure about Envoy TCP proxy but what I disliked in nginx-ingress was that if you set it to TCP proxy then it can't match hostnames anymore, it's more like 8080:namespace/serviceName:port:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-configmap
  namespace: kube-system
data:
  8080: "default/whr:8080"  

Desired behaviour in this case would be that you could still route based on hostnames to different services and ports.

I hope this makes sense.

[davecheney]

Service that ingress points to. LB and Contour would just pass through traffic to the destination, no termination should happen.

Ingress objects are strongly coupled to HTTP. This sounds like a good use case for using a service load balancer directly. Have you tried that? What are the reasons why you decided not to use this approach?

Not sure about Envoy TCP proxy but what I disliked in nginx-ingress was that if you set it to TCP proxy then it can't match hostnames anymore, it's more like 8080:namespace/serviceName:port:

If you're in TCP mode, then i'm guessing there is no HTTP request to inspect to route the traffic. It sounds like you have to dedicate the public IP to forwarding the traffic to only one service, which is very similar to the service load balancer described above.

While I think Envoy can do this, I have some doubts about how to express this in the limitations of the Ingress object.

[davecheney]

@rusenask ping, i'd like to keep this conversation going if you have the time.

[rusenask]

@davecheney hey, apologies.

TLS functionality would be nice so you could mix HTTP and HTTPS (passthrough) services in one ingress.

I know that you could achieve the same thing with multiple service load balancers but having it in on ingress saves you one IP (atleast in GKE) and having external load balancers in GKE costs a bit.

At least in nginx-ingress it have this and I thought it's quite nice as you can have HTTP+HTTPS+GRPC services defined in one ingress. The only downside is that nginx keeps killing long connections. I don't need contour to provision TLS certs as I request them from my backend.

[davecheney]

Thanks for your reply. I'll be honest, I'm not sure I understand what you're asking for, but I do understand supporting the

ingress.kubernetes.io/ssl-passthrough: "true"

annotation. So I'll work on that and we can see if there is anything left to do afterwards.

[davecheney]

The way I think this can be accomplished is to use a tcp proxy filter on the tls listener.

There are a number of caveats with this approach:

• It relies on TLS/SNI for host name routing. You can't do this on normal port 80 without binding to a new port.
• Envoy _must_ do TLS negotiation before making the backend connection. I don't know how to forward the TLS negotiation to the cluster backend, that seems to defeat the SSL spec. The backend service will have a TLS connection, but that TLS connection will be between envoy and itself, not the remote client.

If someone wants to attempt this, I'll mark it for 0.5, but I think the limitations are severe enough that you'd be better off using a service loadbalancer in TCP mode, you're not getting much value by adding Envoy into the mix.

[davecheney]

I'm going to remove the milestone from this as i'm not sure how to implement it within the framework that the k8s Ingress document provides.

[roadrunner]

this is only supported by Haproxy with L4 TCP SNI inspection.

frontend fc_tls
  bind *:443
  mode tcp
  tcp-request inspect-delay 3s
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend abc-tls if { req_ssl_sni -i abc.example.org }
  use_backend xyz-tls if { req_ssl_sni -i xyz.example.org }
  default_backend https-back

currently Envoy doesn't support this feature but there is an old issue still active and @PiotrSikora working on a PR..

[davecheney]

Blocked on envoyproxy/envoy#1843

[glerchundi]

Now that envoyproxy/envoy#1843 was already addressed and released on Envoy 1.7.0, it can be done once #443 gets fixed. Before doing anything and knowing that Contour is taking the use IngressRoute CRD approach (thus not polluting standard ingress with a myriad of annotations), what about something like this?

spec:
  virtualhost:
    fqdn: "www.google.com"
    tls:
      passthrough: true

I don't know if it's even possible but appears to be easily understandable.

[glerchundi]

@davecheney, @rosskukulinski can we push this forward knowing that IngressRoute (project) is already more or less well-established?

[davecheney]

Thanks for the ping. I recently finished the upgrade of Envoy to 1.7.0 for the 0.7 milestone so I think we have all the raw materials needed to do this.

However,

spec:
  virtualhost:
    fqdn: "www.google.com"
    tls:
      passthrough: true

This isn't the only config that would change. K8s services, what envoy calls clusters, are tired implicitly to routes in the ingressroute spec. Some thought about how to change the routes... section of the spect will probably be needed.

[glerchundi]

As it seems to be declared as out-of-scope in #787, when do you think would be the best release/time window to implement this? I'm open to collaborate on this.

[davecheney]

I know you've been waiting a long time for this so I want to try to give you the most complete explanation of the current status.

787 will add TCP port forwarding for TLS connections. They will be decrypted at the Envoy edge and the SNI handshake used to direct the unencrypted traffic to the target pods.

This ticket is a request to add nginx's ssl-passthrough option. Having read the docs for this feature, https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough, I _believe_ that #787 will add the support to Contour to enable this.

However, envoyproxy/envoy#1843, suggests that it is possible to do the SNI handshake, but somehow forward the encrypted traffic to the backend service. From my understanding of the nginx documentation this is _not_ now ssl-passthrough works. But for completeness the nginx docs don't specify if the traffic inside the cluster is decrypted or not.

Given the ambiguity, I forwarding encrypted traffic through Envoy, what I believe this ticket is asking for, is not in scope for Contour 0.8, but obviously doing most of this plumbing will make it possible to add it in the future.

[PiotrSikora]

@davecheney ssl-passthrough is implemented by peeking at the TLS ClientHello message without consuming it (i.e. without performing TLS handshake at the proxy), and simply forwarding encrypted TCP packets between TLS endpoints, in both: Envoy (using TLS Inspector) and NGINX (using ngx_stream_ssl_preread_module).

The only difference between this and #787 (assuming you're using filter chain matching in Envoy) is whether or not tls_context is included in the config.

[davecheney]

@PiotrSikora thanks for confirming.

[glerchundi]

Cannot explain better than how @PiotrSikora did, thanks! And also for the tip that the absence of tls_context makes envoy work as ssl passthrough.

[glerchundi]

@davecheney just to let you know that we're going to start testing this is the next days.

I came across with a patch that should work. Obviously this wouldn't work without the invaluable hint from @PiotrSikora and with your fantastic work on making TCP proxying / forwarding a reality.

Here it is:

From 865201dacdea553f30d1d8bc0f114275c3392bfa Mon Sep 17 00:00:00 2001
From: Gorka Lerchundi Osa <[email protected]>
Date: Wed, 19 Dec 2018 22:02:27 +0100
Subject: [PATCH] tcpproxy: add tls passthrough

---
 internal/contour/listener.go | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/internal/contour/listener.go b/internal/contour/listener.go
index a13caca..1ae9b85 100644
--- a/internal/contour/listener.go
+++ b/internal/contour/listener.go
@@ -16,12 +16,13 @@ package contour
 import (
    "sync"

-   "github.com/envoyproxy/go-control-plane/envoy/api/v2"
+   v2 "github.com/envoyproxy/go-control-plane/envoy/api/v2"
+   "github.com/envoyproxy/go-control-plane/envoy/api/v2/auth"
    "github.com/envoyproxy/go-control-plane/envoy/api/v2/listener"
    "github.com/gogo/protobuf/proto"
    "github.com/heptio/contour/internal/dag"
    "github.com/heptio/contour/internal/envoy"
-   "k8s.io/api/core/v1"
+   v1 "k8s.io/api/core/v1"
 )

 const (
@@ -230,11 +231,6 @@ func (v *listenerVisitor) visit(vertex dag.Vertex) {
        // the listener properly.
        v.http = true
    case *dag.SecureVirtualHost:
-       data := vh.Data()
-       if data == nil {
-           // no secret for this vhost, skip it
-           return
-       }
        filters := []listener.Filter{
            envoy.HTTPConnectionManager(ENVOY_HTTPS_LISTENER, v.httpsAccessLog()),
        }
@@ -246,11 +242,15 @@ func (v *listenerVisitor) visit(vertex dag.Vertex) {
            alpnProtos = nil // do not offer ALPN
        }

+       var tlsContext *auth.DownstreamTlsContext
+       if data := vh.Data(); data != nil {
+           tlsContext = envoy.DownstreamTLSContext(data[v1.TLSCertKey], data[v1.TLSPrivateKeyKey], vh.MinProtoVersion, alpnProtos...)
+       }
        fc := listener.FilterChain{
            FilterChainMatch: &listener.FilterChainMatch{
                ServerNames: []string{vh.Host},
            },
-           TlsContext:    envoy.DownstreamTLSContext(data[v1.TLSCertKey], data[v1.TLSPrivateKeyKey], vh.MinProtoVersion, alpnProtos...),
+           TlsContext:    tlsContext,
            Filters:       filters,
            UseProxyProto: bv(v.UseProxyProto),
        }
-- 
2.20.1

Will tell you something with our conclusions or problems we encounter on the road.

In case someone is interested this is already implemented & published in: docker.io/glerchundi/contour:v0.8.1-cors_tlspassthrough.

[davecheney]

@glerchundi thank you for working on this. I need you to not set patches but signed PRs. Sorry this is a lawyer thing.

[glerchundi]

Of course Dave, will do once I empirically verify it’s something that
works. I included the patch only to show you how it’s being done.

Good night,

On Wed, 19 Dec 2018 at 22:26, Dave Cheney [e-mail] wrote:

@glerchundi https://github.com/glerchundi thank you for working on
this. I need you to not set patches but signed PRs. Sorry this is a lawyer
thing.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/issues/15#issuecomment-448749503, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACIPln-oCqKGjZOd2iKsAtq_3SWXux3xks5u6q8HgaJpZM4QMmnF
.