Ingress-nginx: Wildcard certificate needs to specify every single domain

[pvanderlinden]

NGINX Ingress controller version: 0.15.0
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.3", GitCommit:"d2c7a2bd41036f9474287579a725dc54c904e92d", GitTreeState:"clean", BuildDate:"2018-05-23T00:19:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

What happened:
I'm trying to use a wildcard certificate for ingress, but I have to specify every single domain which I use.

What you expected to happen:
To just match on '*.'

The spec:

  tls:
  - secretName: {{ certificate_name }}-tls
    hosts:
    - '*.{{ domain }}'
    - {{ domain }}
    - www.{{ domain }}
    - app.{{ domain }}
    - api.{{ domain }}

I don't expect the last 3 to be needed, but if I remove those, the ingress will fallback to a self signed certificate.

[muenchdo]

I'm using

  tls:
  - secretName: example-com-tls
    hosts:
    - example.com
    - '*.example.com'

and it works. Maybe you need to reverse the order of hosts?

[pvanderlinden]

Unfortunately that doesn't fix the issue. The only thing I can imagine is that the ingress expects all domains to have 1 dot, e.g. (didn't test that yet):
*.example.com vs *.example.co.uk

[muenchdo]

I just used example.com as an example, I'm actually using domains with more than one dot myself.

[aledbf]

@pvanderlinden just in case, if you have an SSL certificate for *.example.co.uk, that should work ootb for hosts like www.example.co.uk or XXXX.example.co.uk but not for something like demo.app1.example.co.uk

[antoineco]

The controller compares the values in the tls section with the exact host names defined in http rules, but it also tries to guess the best certificate to use in case there is not matching host name.

2 ways to configure a wildcard:

• Add every host name individually
• Omit hosts after the secret name, in which case each host will be checked individually

https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/controller.go#L1082

That's the reason why it works for you @muenchdo, omitting *.example.com would have the same result (but it's good to keep it for documentation).

[pvanderlinden]

Not sure why muenchdo example works, unless he has a wildcard match in the endpoints as well?

[pvanderlinden]

@antoineco Unfortunately that still doesn't work. I omitted hosts, and the ingress still falls back to the self signed certificate.

[antoineco]

Can you show the detail of your wildcard certificate?
openssl x509 -text -noout -in cert.pem

[pvanderlinden]

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:c4:ea:f0:9c:8e:0a:7e:0b:a2:54:cf:2b:34:d0:b1:a5:a2
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Jun 12 17:09:47 2018 GMT
            Not After : Sep 10 17:09:47 2018 GMT
        Subject: CN=*.stg.inflo.ai
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c0:b1:33:46:fc:57:ca:1c:a0:40:84:e7:07:68:
                    aa:78:a9:44:34:45:fc:b2:a2:6f:8e:71:5b:db:d6:
                    62:39:a6:89:60:50:e7:7a:48:51:f5:3b:34:fb:67:
                    9a:5f:e4:8c:d4:03:d2:8b:55:6a:e1:b8:3c:a3:22:
                    1d:aa:ba:b1:9f:46:bd:04:f9:3b:df:25:4a:63:84:
                    a3:87:62:73:70:c4:64:2d:6b:9e:e8:ce:6b:4f:a7:
                    38:83:8f:f6:c8:bb:f6:d6:37:2d:f0:96:9a:76:f2:
                    ec:e6:c1:18:21:8e:f7:ce:33:c6:2b:0c:b3:a3:d2:
                    62:4a:c6:d0:a9:0d:21:79:e5:bb:2a:33:b8:90:42:
                    a9:54:f0:f5:10:45:f3:b9:e6:27:97:b3:a6:a5:cf:
                    33:9e:2f:0a:94:e9:30:34:7a:80:9e:2c:ed:f8:ec:
                    a9:76:f0:c5:6a:71:44:f0:08:f8:72:36:0c:39:79:
                    a8:3e:26:d7:ba:4f:d0:bf:b1:a9:4a:85:e4:31:7f:
                    95:43:6b:3e:38:5b:7d:4d:b2:21:1c:db:4b:fe:3c:
                    12:95:81:24:62:24:71:a4:79:b0:e5:b9:e8:ab:e3:
                    da:64:66:68:96:36:85:79:66:46:a9:b1:eb:08:8e:
                    ed:7f:9a:9a:b5:9c:42:e8:b7:88:c3:01:47:c4:a7:
                    8c:3f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                9C:61:E1:0F:C2:37:51:E1:D2:12:0B:C0:05:8C:7C:B4:58:DF:A5:DF
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:*.stg.inflo.ai, DNS:stg.inflo.ai
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : DB:74:AF:EE:CB:29:EC:B1:FE:CA:3E:71:6D:2C:E5:B9:
                                AA:BB:36:F7:84:71:83:C7:5D:9D:4F:37:B6:1F:BF:64
                    Timestamp : Jun 12 18:09:47.836 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:0F:FF:05:77:CC:35:3B:B6:51:56:E5:D0:
                                3A:46:2F:06:16:DC:C2:A6:CA:C2:34:97:85:88:44:53:
                                5F:5C:5B:EC:02:20:27:89:80:E6:86:D0:2D:95:78:76:
                                B2:D0:90:FA:38:0A:76:F9:4F:0C:E6:A3:29:E4:FF:29:
                                CE:AC:4A:F3:0F:90
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                                6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
                    Timestamp : Jun 12 18:09:47.854 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:90:BC:FE:CA:EA:2E:4B:EB:0B:9F:4E:
                                60:23:71:1D:ED:93:59:E2:89:01:A1:1B:F1:64:8C:2C:
                                33:C8:E0:09:24:02:20:0A:38:B9:2C:39:35:DD:88:34:
                                EF:02:4F:A4:EE:78:56:E5:73:FC:C0:E5:75:EA:46:8B:
                                7C:61:3F:A0:AC:49:67
    Signature Algorithm: sha256WithRSAEncryption
         8c:24:05:5e:59:7a:bc:8b:47:db:e7:02:03:b0:88:14:c5:e3:
         45:bd:5a:8c:64:b9:f2:47:66:0a:2c:36:09:87:72:6e:a6:16:
         9b:cd:4e:57:bb:81:4e:b8:47:21:7d:b6:b3:f7:b8:e9:a3:f6:
         88:85:17:dd:9c:a1:92:6e:d3:d5:44:73:16:13:d5:c4:0f:1a:
         59:b2:04:f0:9f:76:9a:e9:f3:c4:1b:a1:dc:a8:e1:7e:ba:62:
         33:3b:b7:33:6d:78:a3:08:ca:ab:dc:14:10:f1:d2:cc:56:f5:
         de:f9:4d:de:4c:0b:be:a2:90:01:e0:fe:10:93:2d:57:0f:fa:
         8a:f4:b4:a0:23:df:a1:be:74:31:e5:33:d1:09:aa:29:0b:db:
         11:2d:0f:19:94:b3:a7:f0:1f:dc:9b:f5:c0:09:2e:f4:41:c1:
         08:97:28:d3:37:51:43:89:e7:9c:66:49:8c:a8:13:a8:e9:e3:
         ab:27:db:18:ec:7a:dd:bf:1a:81:fc:cf:97:10:68:51:71:d7:
         83:f0:52:5e:21:d3:0e:a7:58:26:b3:b9:98:18:dd:b3:28:22:
         78:66:1a:3c:a4:fe:3d:10:1b:0d:eb:96:d0:1a:38:56:5d:b3:
         ac:98:27:60:d6:af:88:b5:2a:44:fc:27:a8:58:62:ab:76:ac:
         5e:20:5c:67

[aledbf]

@pvanderlinden what's the host in the ingress?

[pvanderlinden]

I was a bit hestitant to post this, but I guess it should be ok:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: infloai
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www.stg.inflo.ai
    http:
      paths:
      - backend:
          serviceName: landing-page
          servicePort: 80
        path: /
  - host: stg.inflo.ai
    http:
      paths:
      - backend:
          serviceName: landing-page
          servicePort: 80
        path: /
  - host: app.stg.inflo.ai
    http:
      paths:
      - backend:
          serviceName: infloai-edit
          servicePort: 80
        path: /
  - host: api.stg.inflo.ai
    http:
      paths:
      - backend:
          serviceName: rest-api
          servicePort: 80
        path: /
  tls:
  - secretName: stg-inflo-ai-tls
    hosts:
    - '*.stg.inflo.ai'
    - stg.inflo.ai
    - www.stg.inflo.ai
    - app.stg.inflo.ai
    - api.stg.inflo.ai

[pvanderlinden]

@antoineco Any idea why it refuses to use the certificate unless I specify every single host?

[antoineco]

No, the host names look correct, I'm a bit puzzled.

[pvanderlinden]

Is there anything I can do to figure out this problem? I haven't found anything strange in the logs of the nginx container.

[antoineco]

@aledbf The logic only compares the host name with the certificate's CN, however @pvanderlinden's certificate has CN:[]string{"*.stg.inflo.ai"}.

1. We need to take the wildcard into account
2. We should preferable use the SAN over the CN

x509.Certificate.VerifyHostname would do that.

[antoineco]

~@pvanderlinden here is a rebuild of v0.16.1 including the patch above: antoineco/nginx-ingress-controller-amd64:0.16.1~

Thanks for reporting this 🙌

[pvanderlinden]

Looks like I can't access that repository, I will wait till the the official build. Thanks for fixing!

[aledbf]

@pvanderlinden please update to 0.16.2 (contains #2698)

[pvanderlinden]

Thanks for letting me know. This works!

[MahdiZareie]

I'm using 0.16.2 and I guess I have the same problem:
I'm getting this error from logs:

SSL certificate "default/roozameh-net-tls" does not contain a Common Name or Subject Alternative Name for server "foo.bar.baz.biz.roozameh.net": x509: certificate is valid for *.roozameh.net, roozameh.net, not foo.bar.baz.biz.roozameh.net

and this is my ingress configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-world-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
    - hosts:
      - "*.roozameh.net"
      - "foo.bar.baz.biz.roozameh.net"
      secretName: roozameh-net-tls
  rules:
  - host: foo.bar.baz.biz.roozameh.net
    http:
      paths:
      - path: /
        backend:
          serviceName: hello-world-svc
          servicePort: 8080

the *.roozameh.net should match my sub-sub domain, but ingress complain about it.
Am I doing something wrong?
UPDATE:
this is my certificate details:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:cf:16:68:a6:ac:f3:0b:47:d0:ee:34:44:16:71:0c:cf:ce
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Jul 14 08:13:08 2018 GMT
            Not After : Oct 12 08:13:08 2018 GMT
        Subject: CN=*.roozameh.net
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:df:35:1a:6b:46:c6:ba:73:e7:dc:26:ab:64:26:
                    2b:95:97:b5:6f:fd:5a:dd:00:c6:92:d0:89:05:d3:
                    46:e8:d5:8f:c3:a1:7a:81:00:fd:90:fd:58:39:35:
                    ad:be:48:5c:ef:a1:5e:11:27:17:bc:76:d3:d6:5c:
                    f0:1c:97:0b:f1:7a:0e:7f:11:5c:65:0d:39:6b:84:
                    8c:96:7d:c6:24:4c:76:23:ee:b0:28:8c:fb:cd:b6:
                    b0:8a:0d:08:43:79:e5:2b:8f:dc:9e:2e:97:ec:7b:
                    17:5a:72:cc:9f:50:2d:86:f1:e3:c9:f8:da:c7:b3:
                    19:65:db:d0:f5:15:f0:c1:ce:c5:f1:76:26:d8:cb:
                    19:3c:ce:dc:eb:59:e3:c2:a3:ba:b1:96:72:1c:3c:
                    2d:b8:53:ef:75:fd:d1:53:3f:39:42:61:72:61:59:
                    f4:cc:d0:81:f2:2f:cc:15:5e:a0:9a:ce:6a:29:04:
                    e3:e7:c2:0a:52:d4:cd:ee:39:45:d7:52:9e:b6:47:
                    5e:2d:92:cc:5d:a0:c5:b2:1a:17:f7:25:68:21:60:
                    fe:1e:f4:c8:59:5b:08:76:3d:fb:91:4e:e2:28:bc:
                    b2:61:1d:60:c4:16:db:55:43:af:3c:14:32:c5:39:
                    3c:4f:04:84:49:71:b1:0c:2e:1e:49:0e:78:e4:5c:
                    59:57
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                7E:11:E0:CF:53:20:A1:EB:70:D4:EA:AF:11:D6:11:1F:DA:50:0A:75
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access:
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name:
                DNS:*.roozameh.net, DNS:roozameh.net
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org
                  User Notice:
                    Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                                6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
                    Timestamp : Jul 14 09:13:08.896 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:F0:B2:9E:EB:E3:48:F0:7B:B3:D8:13:
                                DF:C2:F9:51:28:3A:A0:CA:96:CB:09:18:67:1B:32:9F:
                                9E:40:36:CF:BF:02:20:4B:1A:91:89:97:22:47:E9:67:
                                CC:0E:FA:D7:B7:87:F5:32:1D:C6:59:67:6A:B0:87:F4:
                                E5:87:D4:B2:38:B7:F2
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : 55:81:D4:C2:16:90:36:01:4A:EA:0B:9B:57:3C:53:F0:
                                C0:E4:38:78:70:25:08:17:2F:A3:AA:1D:07:13:D3:0C
                    Timestamp : Jul 14 09:13:09.517 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:B3:04:DB:C1:AB:6A:0F:D6:43:E7:DD:
                                11:BE:FA:70:CB:DE:89:7F:BA:4B:EA:8A:DC:A2:DC:9D:
                                1E:1E:DC:67:F1:02:21:00:BB:B5:D3:A8:05:3D:41:2F:
                                91:29:48:C4:81:05:F0:E9:D5:F9:C1:FF:BB:27:C6:B6:
                                7E:BB:74:B0:9D:BA:62:7A
    Signature Algorithm: sha256WithRSAEncryption
         03:43:18:13:72:4f:af:b6:6f:15:71:fc:11:55:84:ee:cb:ea:
         92:7e:05:e9:a4:a4:83:af:98:ca:f9:5e:32:49:87:46:f8:ff:
         47:34:3f:f4:9c:bd:b6:46:d3:a6:ba:3e:c2:a4:21:48:b6:b3:
         e8:09:f1:2f:2d:82:af:bc:53:1a:4a:e5:7e:d8:8f:52:45:33:
         cb:1b:16:52:44:a7:92:ff:03:7d:72:d2:87:26:82:a5:fb:69:
         48:10:22:c6:ca:47:60:be:20:45:de:77:28:2d:11:21:4f:40:
         1b:92:28:0c:29:01:ec:43:c8:16:86:46:19:95:5d:3b:aa:19:
         fb:ad:ca:cc:db:0b:ab:6d:b1:7c:75:34:45:4f:70:3d:22:8f:
         ba:7d:07:7a:e8:76:2a:1d:2b:9f:fc:d4:77:ad:a7:4e:17:01:
         b4:1e:ee:81:50:5e:a0:0e:6d:4d:3f:1e:66:e2:88:8b:4d:a2:
         a8:37:fb:6b:ca:45:e7:e4:91:65:77:2f:89:b3:e9:67:6e:3b:
         de:14:79:d8:75:9a:ff:7d:c8:b5:e7:e7:d8:98:21:6c:39:62:
         32:6f:0f:d9:12:14:44:41:8b:f5:c8:70:8c:5d:de:ef:43:ee:
         f4:02:92:c5:ef:dc:86:07:3a:ad:63:86:8a:77:19:1c:7e:cf:
         15:a5:c4:7f

[pieterlange]

Wildcards are only valid for a single DNS label. So biz.roozameh.net would match *.roozameh.net but foo.bar.baz.biz.roozameh.net and baz.biz.roozameh.net don't.

[PatrickGeyer]

I need a solution that would let me set a tls to be used by default, as my web app is run from multiple domains as clients white-label it and host it on their own sites. At the moment I just specify every host, but this is cumbersome. The certificate is a SAN, which means the one certificate works for all the domains, so I just want to use it by default.
Anyone know how to go about this?

[pieterlange]

You could use --default-ssl-certificate (https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/)

[TheAifam5]

@pieterlange but the certificate DNS *.*.docker.zz also does not work for a domain like <service>.<namespace>.docker.zz.

[alt_names]
DNS.1 = docker.zz
DNS.2 = *.docker.zz
DNS.3 = *.*.docker.zz

[antoineco]

@TheAifam5 your wildcard can only cover 1 subdomain level (*.dom, not *.*.dom). If you need one wildcard per namespace you need to have multiple certificates for *.default.dom, *.myapp.dom, etc.

[TheAifam5]

You're right: https://tools.ietf.org/html/rfc2818#section-3.1

Thanks :)

Can I somehow define for a specific namespace to use the default (saved in tls secret) certificate?

[tegk]

@pvanderlinden did you find a solution for this? I have the same problem.