Envoy: Mututal TLS: x-forwarded-client-cert header not being forwarded to destination

[arianmotamedi]

I have the following k8s infrastructure setup to allow mutual tls between proxy side cars:

      http:20000 (egress)                        https:443                    https:10000 (ingress)
App1 ---------------------> App1's Envoy --------------------------> kubedns ------------------------> App2's Envoy ---------> App2

And this is my static yaml configuration that I use for both App1 and App2's envoys (cluster host addresses are k8s dns names):

static_resources:
  listeners:
  - name: ingress
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
      tls_context:
        require_client_certificate: true
        common_tls_context:
          tls_certificates:
            certificate_chain: { filename: /mnt/certs/server/server.pem }
            private_key: { filename: /mnt/certs/server/server.key.pem }
      filters:
      - name: envoy.http_connection_manager
        config:
          server_name: my-service
          stat_prefix: ingress_http
          codec_type: AUTO
          forward_client_cert_details: ALWAYS_FORWARD_ONLY
          use_remote_address: true
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              require_tls: ALL
              routes:
              - match: { prefix: "/" }
                route: { cluster: my-service }
          http_filters:
          - name: envoy.router
  - name: egress
    address:
      socket_address: { address: 127.0.0.1, port_value: 20000 }
    filter_chains:
      filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: egress_http
          codec_type: AUTO
          forward_client_cert_details: ALWAYS_FORWARD_ONLY
          route_config:
            name: ext_route
            virtual_hosts:
            - name: ext_services
              domains: ["*"]
              routes:
              - match: { prefix: "/app1/" }
                route: 
                  cluster: app1
                  prefix_rewrite: "/"
              - match: { prefix: "/app2/" }
                route:
                  cluster: app2
                  prefix_rewrite: "/"
          http_filters:
          - name: envoy.router
  clusters:
  - name: my-service
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
        address: 127.0.0.1
        port_value: 8080
  - name: app1
    connect_timeout: 1s
    type: STRICT_DNS
    hosts:
    - socket_address:
        address: app1
        port_value: 443
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: /mnt/certs/client/client.pem }
          private_key: { filename: /mnt/certs/client/client.key.pem }
  - name: app2
    connect_timeout: 1s
    type: STRICT_DNS
    hosts:
    - socket_address:
        address: app2
        port_value: 443
    tls_context:
      common_tls_context:
        tls_certificates:
        - certificate_chain: { filename: /mnt/certs/client/client.pem }
          private_key: { filename: /mnt/certs/client/client.key.pem }

And my k8s deployment spec (the service listens on port 443 and forwards inbound traffic to port 10000 of the destination container, which is the ingress port for Envoy)

apiVersion: v1
kind: Service
metadata:
  name: app1
  namespace: envoy-test
  labels:
    app: app1
spec:
  ports:
  - port: 443
    targetPort: 10000
    name: https
  selector:
    app: app1

App2 has an endpoint (/dump) that prints all headers in the request object. My expectation of this was when App1 calls App2/dump, it would print the x-forwarded-client-cert header, but it's missing from the request:

curl localhost:20000/app2/dump
{"Accept":["*/*"],"Content-Length":["0"],"User-Agent":["curl/7.60.0"],"X-Envoy-Expected-Rq-Timeout-Ms":["15000"],"X-Envoy-Internal":["true"],"X-Envoy-Original-Path":["/app2/dump"],"X-Forwarded-For":["10.224.100.164"],"X-Forwarded-Proto":["https"],"X-Request-Id":["9590796f-1ba9-4294-a754-34b5e0d2c317"]}

This request was made from App1's container. I then exec'ed into App1's envoy container and tried to call App2 directly without going through the source proxy:

curl -k -v https://app2/dump
*   Trying 172.20.111.247...
* TCP_NODELAY set
* Connected to app2 (172.20.111.247) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=proxy.my-apps.com
*  start date: Jun 15 18:05:57 2018 GMT
*  expire date: Sep 13 18:05:57 2018 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /dump HTTP/1.1
> Host: app2
> User-Agent: curl/7.61.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< date: Wed, 18 Jul 2018 16:35:20 GMT
< content-length: 264
< x-envoy-upstream-service-time: 0
< server: my-service
< 
* Connection #0 to host app2 left intact
{"Accept":["*/*"],"Content-Length":["0"],"User-Agent":["curl/7.61.0"],"X-Envoy-Expected-Rq-Timeout-Ms":["15000"],"X-Envoy-Internal":["true"],"X-Forwarded-For":["10.224.100.164"],"X-Forwarded-Proto":["https"],"X-Request-Id":["178f3ced-07e8-4a7a-8ddd-b01fd9bb2560"]}

this shows that the connection is in fact over https (and subject: CN=proxy.my-apps.com shows that it's loading the correct server certificate), but if you notice, I didn't specify a client certificate, and even though I had set require_client_certificate: true in my ingress listener, the connection still didn't get terminated (this seems like a bug). But, even after explicitly specifying a client certificate through curl, the header is still not getting forwarded:

curl -k -v https://app2/dump --cert mnt/certs/client/client.pem --key mnt/certs/client/client.key.pem
*   Trying 172.20.111.247...
* TCP_NODELAY set
* Connected to app2 (172.20.111.247) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=proxy.my-apps.com
*  start date: Jun 15 18:05:57 2018 GMT
*  expire date: Sep 13 18:05:57 2018 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /dump HTTP/1.1
> Host: app2
> User-Agent: curl/7.61.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< date: Wed, 18 Jul 2018 16:40:45 GMT
< content-length: 264
< x-envoy-upstream-service-time: 0
< server: my-service
< 
* Connection #0 to host app2 left intact
{"Accept":["*/*"],"Content-Length":["0"],"User-Agent":["curl/7.61.0"],"X-Envoy-Expected-Rq-Timeout-Ms":["15000"],"X-Envoy-Internal":["true"],"X-Forwarded-For":["10.224.100.164"],"X-Forwarded-Proto":["https"],"X-Request-Id":["17b8282c-7833-4772-aec5-08f2785763d0"]}

I have been trying to troubleshoot this without any success for the past few days. I'm not sure what I'm missing from my Envoy configuration... any help would be truly appreciated. I'd be happy to provide more info if needed.

[arianmotamedi]

It turned out a valid validation_context with a CA chain was needed on the ingress listener for client certificate header to be passed along to upstream service and also for require_client_certificate: true to be taken into account. This should probably be mentioned in the documentation :)

[justin-yen]

Does the above still hold true for istio 1.4.0? Is there no way to require a client certificate at the ingress gateway but not validate it, and simply pass it through?

[wlhee]

I wonder does the App1's Envoy automatically attach the client cert as xfcc header ?

[chrismoos]

I know this is closed, but if anyone is looking, you can get Envoy to pass through client certificate details (via XFCC header) without performing any validation on the client certificate with the following in common_tls_context:

validation_context:
  trust_chain_verification: ACCEPT_UNTRUSTED

The above will let you establish a TLS session with any client certificate, letting an upstream server consume the XFCC header and perform any authentication/verification instead.