Ingress-nginx: auth-tls-pass-certificate-to-upstream does not work with https

Created on 4 Dec 2018 · 21Comments · Source: kubernetes/ingress-nginx

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.21.0

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-27T01:14:37Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

What happened:

When adding auth-tls-pass-certificate-to-upstream: true to an ingress resource, the client certificate passed to the ingress controller is not forwarded to the backend pod.

What you expected to happen:

The backend pod should receive the client certificate.

How to reproduce it (as minimally and precisely as possible):

Start an https server expecting mtls
Create an ingress resource, such as the following, that points to the mtls server's service

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mtls-sample
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "https"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
spec:
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: mtls-svc
              servicePort: 443

Send a request to the ingress controller, such as the following.

# curl -L --cert 4_client/certs/localhost.cert.pem --key 4_client/private/localhost.key.pem https://172.17.0.11:443/hello -k

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.15.6</center>
</body>
</html>

View the pod logs (assuming go app) to verify missing client cert.

2018/12/04 17:42:29 http: TLS handshake error from 172.17.0.11:41922: tls: client didn't provide a certificate

Send same curl directly to pod service and verify mtls succeeds.

# curl -L --cert 4_client/certs/localhost.cert.pem --key 4_client/private/localhost.key.pem https://10.102.202.95:443/hello -k

Hello World/

Anything else we need to know:

Unless i'm misunderstanding the annotation, I'd expect the client cert to be passed on to the upstream pod.

Source

joshrosso

Most helpful comment

Again, still interested! :)

/remove-lifecycle stale

baurmatt on 14 Oct 2019

👍6

All 21 comments

Alright, after digging deeper, I'm finding the issue to be more around standardization of passing client certificates in headers rather than my initial theory, the nginx-ingress-controller not passing the client cert.

I've found nginx is passing the client cert to the backend pod in the Ssl-client-certificate header.

It seems for projects like Envoy, there's been lots of discussion around how to accomplish this, in their case they went with x-forwarded-client-cert. And some suggestions on stackoverflow suggest using headers like x-ssl-cert.

I suggest we provide the functionality to specify the header key for which the client cert will be forwarded in. Something such as:

nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream-header: "x-forwarded-client-cert"

To ensure compatibility with upstream servers / pods.

Let me know your thoughts.

joshrosso on 4 Dec 2018

I suggest we provide the functionality to specify the header key for which the client cert will be forwarded in. Something such as:

That makes sense. We already do this for another header https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#forwarded-for-header
That said, this should be a global value instead of a new annotation, at least as a first step.

Edit: just in case, this does not means the header will be compatible with envoy (http://nginx.org/en/docs/http/ngx_http_ssl_module.html != https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#x-forwarded-client-cert)

aledbf on 4 Dec 2018

this should be a global value instead of a new annotation,

I can take a stab at this. Can you elaborate on the meaning of global value @aledbf ?

this does not means the header will be compatible with envoy

100% understood.

joshrosso on 4 Dec 2018

Can you elaborate on the meaning of global value @aledbf ?

Sorry, we use a configmap that configures global values https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/config/config.go#L431 with defaults https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/config/config.go#L586

aledbf on 4 Dec 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 4 Mar 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 4 Apr 2019

Any update on this? :) I really like to see this implemented!

/remove-lifecycle rotten

baurmatt on 16 Apr 2019

👍5

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 15 Jul 2019

Still interested! :)

/remove-lifecycle stale

baurmatt on 15 Jul 2019

👍2

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 13 Oct 2019

Again, still interested! :)

/remove-lifecycle stale

baurmatt on 14 Oct 2019

👍6

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 12 Jan 2020

Can't wait to get this! ;)

/remove-lifecycle stale

baurmatt on 13 Jan 2020

👍3

did anyone figure this out, we have the same situation, had to spend a few days debugging before landing here.

akashvyom on 24 Feb 2020

Any update on this? We have a same situation like this.

manpatel3107 on 9 Mar 2020

Hello,

I run into this thread a couple of days ago.

It seems that there is a configuration snippet annotation (https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#configuration-snippet) that lets you define the HTTP header in which the client certificate will be inserted. The following annotation worked for me:

nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

Of course you can replace the 'X-SSL-CERT' with the name of your desired header.

IoakeimSamarasGR on 18 May 2020

@IoakeimSamarasGR Could you please share your whole annotation?

I am using the following annotation without success

annotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/affinity: "cookie"
      nginx.ingress.kubernetes.io/session-cookie-name: "MYSERVICE"
      nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
      nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
      #nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
      #nginx.ingress.kubernetes.io/auth-tls-verify-client: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

rdoering on 2 Jul 2020

@rdoering Here are the annotations that worked for me:

annotations:
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/{{ .Values.ingress.caTls.caSecretName }}
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
      proxy_ssl_name "mydemo.demo.com";
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-secret: default/{{ .Values.ingress.caTls.caSecretName }}
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "1"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /

IoakeimSamarasGR on 6 Jul 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 4 Oct 2020

/remove-lifecycle stale

frittentheke on 10 Oct 2020

@IoakeimSamarasGR Could you please share your whole annotation?

I am using the following annotation without success

annotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/affinity: "cookie"
      nginx.ingress.kubernetes.io/session-cookie-name: "MYSERVICE"
      nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
      nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
      #nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
      #nginx.ingress.kubernetes.io/auth-tls-verify-client: "true"
      nginx.ingress.kubernetes.io/configuration-snippet: |
        proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

I gave it a try with kubernetes v1.18.8 and k8s.gcr.io/ingress-nginx/controller:v0.41.2
(reference: https://github.com/kubernetes/ingress-nginx/issues/2922 "Unable to use "ssl_verify_client optional_no_ca" with the current implementation of "auth-tls-verify-client" annotation")

This combination of two configs set the client certificates.

    nginx.ingress.kubernetes.io/server-snippet: |
      ssl_verify_client optional_no_ca;
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header SSL_CLIENT_CERT $ssl_client_escaped_cert;

As far as I tested, nginx.ingress.kubernetes.io/auth-tls-verify-client: "optional_no_ca" did not work. nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true" was not necessary.