Envoy: SNI-based routing without TLS termination

[PiotrSikora]

Pre-read TLS ClientHello and extract requested SNI, which then can be used for routing, blocked on #95.

cc @louiscryan

[louiscryan]

@rshriram FYI

Useful for end-to-end mutual TLS where traffic is routed through edge without TLS termination.

[rshriram]

Ooh.. interesting.

[htuch]

@PiotrSikora can you elaborate for the non-Istio folks? Why is the client in this scenario presenting a ClientHello if there is no TLS termination?

[PiotrSikora]

The idea is that the client establishes TLS connection "directly" with the backend service, without TLS termination at the proxy, i.e. Envoy is simply doing L4 proxying of TCP packets.

However, instead of making routing decisions only based on the listening and/or destination IP:port, we want to take advantage of the fact that the ClientHello (i.e. first message in the TLS handshake) is unencrypted and contains useful information (i.e. ALPN and SNI) to make routing decisions based on that as well.

e.g.

"listeners": [
  {
    "address": "tcp://0.0.0.0:443",
    "filters": [
      {
        "name": "tcp_proxy",
        "config": {
          "stat_prefix": "smart_tcp",
          "route_config": {
            "routes": [
              {
                "domains": [ "google.com", "*.google.com" ],
                "cluster": "service_google"
              },
              {
                "domains": [ "*.appspot.com" ],
                "cluster": "service_app_engine"
              }
            ]
          }
        }
      }
    ]
  }
]

In such scenario, neither the client nor the backend has to trust the proxy, which at this point is just another hop.

[htuch]

Yes, this makes sense. Our team also has a need for being able to interpret the initial TCP stream prefix when proxying to extract out additional info, we should sync about this.

[vadimeisenbergibm]

@PiotrSikora @mattklein123 any estimation when this feature will be implemented?

[PiotrSikora]

@vadimeisenbergibm I'm working on tests for this right now, PR should be out "soon".

[vadimeisenbergibm]

@PiotrSikora glad to hear it, good news!

[evantorrie]

Also interested in this

[sureshvis]

very much interested. We do this today using Apache Traffic Server proxy.

[mastersingh24]

@PiotrSikora - are you planning on supporting both SNI and ALPN?

[PiotrSikora]

@mastersingh24 both SNI and ALPN are extracted from the ClientHello and available to filters and filter chain selector, but the code I have right now doesn't make decisions based on ALPN.

[mpuncel]

any updates on this? This feature is very useful!

[PiotrSikora]

Sorry, I've got side-tracked with unrelated bug hunting, but I've started working on this again today, so the PR should be out "soon" (the code for this & the rewrite of filter match selection logic has been completed for a while, but I'm still missing a few tests).

[gtie]

:+1: :eyes:

[davidamin]

Is there an example configuration for this functionality available? I found a config at How to setup SNI but it includes a tls_context in each match, is it possible to do the routing without that configured?

[PiotrSikora]

@davidamin yes, just remove tls_context.

[tortuoise]

I could do with a little help with the setup for SNI based routing without TLS termination. I've got the SNI setup working with the TLS termination but can't get the tcp_proxy filter to work for sni routing without tls termination.

This config seems to work for SNI routing (though oddly I can only use it with v2.HttpConnectionManager and not v3) :

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    listener_filters:
    - name: "envoy.filters.listener.tls_inspector"
      typed_config: {}
    filter_chains:
    - filter_chain_match:
        server_names: ["www.example.com"]
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext  
          common_tls_context:
            tls_certificates:
            - certificate_chain: { filename: "/etc/envoy/www-crt.pem" }
              private_key: { filename: "/etc/envoy/www-key.pem" }
      filters:
      - name: envoy.filters.network.http_connection_manager # envoy.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager 
         stat_prefix: ingress_http
         access_log:
          - name: envoy.file_access_log
            config:
              path: "/var/log/envoy_access.log"
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match: { prefix: "/" }
                route: { cluster: service1 }
          http_filters:
          - name: envoy.router
    - filter_chain_match:
        server_names: ["api.example.com"]
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext  
          common_tls_context:
            tls_certificates:
            - certificate_chain: { filename: "/etc/envoy/api-crt.pem" }
              private_key: { filename: "/etc/envoy/api-key.pem" }
      filters:
      - name: envoy.filters.network.http_connection_manager # envoy.http_connection_manager 
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager           
          stat_prefix: ingress_http
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/var/log/envoy_access.log"
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match: { prefix: "/" }
                route: { cluster: service2 }
          http_filters:
          - name: envoy.router
    - filter_chain_match:

  clusters:
  - name: service1
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    http2_protocol_options: {}
    load_assignment:
      cluster_name: service1
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service1
                port_value: 8000
   - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    #http2_protocol_options: {}
    load_assignment:
      cluster_name: service2
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: service2
                port_value: 8000
admin:
  access_log_path: "/var/log/envoy_admin.log" 
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

But what's wrong with the below additions to the config?

....
- filter_chain_match:
        server_names: ["sub.example.com"]
  filters:
      - name: envoy.filters.network.tcp_proxy 
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          cluster: passthrough

 clusters:
...
  - name: passthrough
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: passthrough
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: passthrough
                port_value: 8000
...

The existing filter_chain_matches still work but the new one doesn't. The envoy access log seems to route the connection properly but https clients report the connection as being reset.

[2020-05-26T15:54:55.128Z] "- - -" 0 UF,URX 0 0 0 - "-" "-" "-" "-" "172.18.0.6:8000"

[PiotrSikora]

@tortuoise what's running on 172.18.0.6:8000? Is that a TLS server (please remember that connections are not TLS-terminated by Envoy)? URX means that the connection to upstream failed, so the filter chain works, but Envoy cannot connect to the upstream.

[tortuoise]

172.18.0.6:8000 has a tls server running in a separate container (part of same network) with envoy as sidecar. The envoy sidecar config also uses a tcp_proxy filter which finally connects to a cluster with the tls server. This is likely where my error is because I see nothing in the access logs for this envoy.

listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
          cluster: https_service
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/var/log/envoy_access.log"
  clusters:
  - name: https_service
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: https_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8030
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext
        sni: sub.example.com

Alternatively, I also tried to have the tls server running as a local cluster in the same container as the edge envoy (as per the config pasted in previous comment - not the one above). This also doesn't work as expected.

...
- name: local_service
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: local_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8030
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext       
      sni: sub.example.com

[tortuoise]

Ok got this working with the local_service without the transport_socket following @PiotrSikora's reminder that envoy doesn't tls-terminate the connection. It only works with envoy.config.filter.network.tcp_proxy.v2.TcpProxy and not with envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy. Is this expected?

[sumeetoc]

@tortuoise can you please help me with the final configuration file of yours? I seems to have the similar requirement