Ingress-nginx: fake ingress.local cert on a dual tls ingress with wildcard cert. An identical setup on a different cluster works.

[simo-eskalera]

BUG REPORT

NGINX Ingress controller version: nginx-ingress-1.6.17

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.10", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-04-12T22:59:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• GKE
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:

$ helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

• Others:
  Cluster running version 1.12.7-gke.10 with a single node running on Ubuntu

What happened:

I have a dual certificate ingress. The tls section has 2 entries. One for app.dukketta.com and a wildcard certificate for *.app.dukketta.com. Only the former works. Any request matching *.app.dukketta.com returns the fake certificate. FYI we have an identical setup on another cluster for dev environment and both certs are properly served for app.dev.dukketta.com and *.app.dev.dukketta.com. The only difference is the nodes on the dev cluster are running on gke's optimized coreos and the pods and ingress live in the default namespace.

Dev Cluster:

• Ingress controller in kube-system namespace
• Services, ingress and secrets live in default namespace
• Cert config for app.dev.dukketta.com work!
• Cert config for *.app.dev.dukketta.com work!

Prod Cluster:

• Ingress controller in kube-system namespace
• Services, ingress and secrets live in production namespace
• Cert config for app.dukketta.com work!
• Cert config for .app.dukketta.com *doesn't work. returns fake cert

What you expected to happen:
The wildcard certificate for *.app.dukketta.com would be sent back to the client.

How to reproduce it (as minimally and precisely as possible):
Here's a list of tests to verify the certs:

(PROD) app.dukketta.com

openssl s_client -connect app.dukketta.com:443 -servername app.dukketta.com -showcerts | grep "Certificate chain" -C 2

reports correctly

Certificate chain
 0 s:/CN=app.dukketta.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

(PROD) test.app.dukketta.com

openssl s_client -connect test.app.dukketta.com:443 -servername test.app.dukketta.com -showcerts | grep "Certificate chain" -C 2

reports the fake cert : ❌

Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate

The similar dev setup from another cluster works:
(DEV) app.dev.dukketta.com

openssl s_client -connect app.dev.dukketta.com:443 -servername app.dev.dukketta.com -showcerts | grep "Certificate chain" -C 2
depth=0 CN = app.dev.dukketta.com

reports correctly

Certificate chain
 0 s:/CN=app.dev.dukketta.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

(DEV) test.app.dev.dukketta.com

openssl s_client -connect test.app.dev.dukketta.com:443 -servername test.app.dev.dukketta.com -showcerts | grep "Certificate chain" -C 2

reports correctly

Certificate chain
 0 s:/CN=*.app.dev.dukketta.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3

Anything else we need to know:

Ingress controller installed as a helm chart:

helm install stable/nginx-ingress --name nginx-ingress --namespace kube-system

Content of the ingress applied to the production namespace

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dukketta-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/server-alias: "*.app.dukketta.com"
spec:
  rules:
  - host: app.dukketta.com
    http:
      paths:
      - path: /api
        backend:
          serviceName: apiserver
          servicePort: 80
      - path: /
        backend:
          serviceName: dukketta-ui
          servicePort: 80
  tls:
  - secretName: prod-app-tls-cert
    hosts:
    - app.dukketta.com
  - secretName: prod-wildcard-app-tls-cert
    hosts:
    - "*.app.dukketta.com"

Secrets generated using letsencrypt certbot and applied this way:

# For app.dukketta.com
kubectl create secret tls prod-app-tls-cert --key /etc/letsencrypt/archive/app.dukketta.com-0001/privkey1.pem --cert /etc/letsencrypt/archive/app.dukketta.com-0001/cert1.pem -n production

#For *.app.dukketta.com
kubectl create secret tls prod-wildcard-app-tls-cert --key /etc/letsencrypt/archive/app.dukketta.com-0002/privkey2.pem  --cert /etc/letsencrypt/archive/app.dukketta.com-0002/cert2.pem -n production

Result of describe ingress

$ kubectl describe ing dukketta-ingress -n production

Name:             dukketta-ingress
Namespace:        production
Address:          104.196.38.81
Default backend:  default-http-backend:80 (10.28.0.11:8080)
TLS:
  prod-app-tls-cert terminates app.dukketta.com
  prod-wildcard-app-tls-cert terminates *.app.dukketta.com
Rules:
  Host              Path  Backends
  ----              ----  --------
  app.dukketta.com
                    /api   apiserver:80 (<none>)
                    /      dukketta-ui:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/server-alias":"*.app.dukketta.com"},"name":"dukketta-ingress","namespace":"production"},"spec":{"rules":[{"host":"app.dukketta.com","http":{"paths":[{"backend":{"serviceName":"apiserver","servicePort":80},"path":"/api"},{"backend":{"serviceName":"dukketta-ui","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["app.dukketta.com"],"secretName":"prod-app-tls-cert"},{"hosts":["*.app.dukketta.com"],"secretName":"prod-wildcard-app-tls-cert"}]}}

  kubernetes.io/ingress.class:               nginx
  nginx.ingress.kubernetes.io/server-alias:  *.app.dukketta.com
Events:
  Type    Reason  Age                  From                      Message
  ----    ------  ----                 ----                      -------
  Normal  UPDATE  3m53s (x3 over 10h)  nginx-ingress-controller  Ingress production/dukketta-ingress

Result of describe secrets

$ kubectl get secrets -n production

NAME                         TYPE                                  DATA   AGE
default-token-79vvb          kubernetes.io/service-account-token   3      12d
prod-app-tls-cert            kubernetes.io/tls                     2      11h
prod-wildcard-app-tls-cert   kubernetes.io/tls                     2      10h

Result of describe ingress controller

$ kubectl describe svc nginx-ingress-controller -n kube-system

Name:                     nginx-ingress-controller
Namespace:                kube-system
Labels:                   app=nginx-ingress
                          chart=nginx-ingress-1.6.17
                          component=controller
                          heritage=Tiller
                          release=nginx-ingress
Annotations:              <none>
Selector:                 app=nginx-ingress,component=controller,release=nginx-ingress
Type:                     LoadBalancer
IP:                       10.0.3.26
LoadBalancer Ingress:     35.243.160.84
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32656/TCP
Endpoints:                10.28.0.34:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31543/TCP
Endpoints:                10.28.0.34:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

[simo-eskalera]

Log from the nginx-ingress-controller pod:

NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0618 13:19:55.847444       6 flags.go:185] Watching for Ingress class: nginx
W0618 13:19:55.848254       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0618 13:19:55.876305       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0618 13:19:55.876829       6 main.go:205] Creating API client for https://10.0.0.1:443
I0618 13:19:55.897656       6 main.go:249] Running in Kubernetes cluster version v1.12+ (v1.12.7-gke.10) - git (clean) commit 8d9b8641e72cf7c96efa61421e87f96387242ba1 - platform linux/amd64
I0618 13:19:55.905475       6 main.go:102] Validated kube-system/nginx-ingress-default-backend as the default backend.
I0618 13:19:56.389669       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0618 13:19:56.411204       6 store.go:613] Unexpected error reading configuration configmap: configmaps "nginx-ingress-controller" not found
I0618 13:19:56.451982       6 nginx.go:265] Starting NGINX Ingress controller
I0618 13:19:57.564730       6 backend_ssl.go:68] Adding Secret "production/prod-app-tls-cert" to the local store
I0618 13:19:57.565712       6 backend_ssl.go:68] Adding Secret "production/prod-wildcard-app-tls-cert" to the local store
I0618 13:19:57.565949       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"dukketta-ingress", UID:"49684f2f-912e-11e9-87d5-42010af00205", APIVersion:"extensions/v1beta1", ResourceVersion:"3033561", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress production/dukketta-ingress
I0618 13:19:57.659581       6 nginx.go:311] Starting NGINX process
I0618 13:19:57.660017       6 leaderelection.go:217] attempting to acquire leader lease  kube-system/ingress-controller-leader-nginx...
I0618 13:19:57.660832       6 controller.go:170] Configuration changes detected, backend reload required.
I0618 13:19:57.669413       6 status.go:86] new leader elected: nginx-ingress-controller-858cb497b6-hqc69
I0618 13:19:57.959762       6 controller.go:188] Backend successfully reloaded.
I0618 13:19:57.960163       6 controller.go:202] Initial sync, sleeping for 1 second.
[18/Jun/2019:13:19:58 +0000]TCP200000.000
[18/Jun/2019:13:20:06 +0000]TCP200000.000
I0618 13:20:41.176557       6 leaderelection.go:227] successfully acquired lease kube-system/ingress-controller-leader-nginx
I0618 13:20:41.176897       6 status.go:86] new leader elected: nginx-ingress-controller-858cb497b6-z5vcn
I0618 13:20:41.188776       6 status.go:295] updating Ingress production/dukketta-ingress status from [] to [{104.196.38.81 }]
I0618 13:20:41.193578       6 backend_ssl.go:60] Updating Secret "production/prod-app-tls-cert" in the local store
I0618 13:20:41.193919       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"dukketta-ingress", UID:"49684f2f-912e-11e9-87d5-42010af00205", APIVersion:"extensions/v1beta1", ResourceVersion:"3033713", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress production/dukketta-ingress

[simo-eskalera]

NVM, I found the issue. Dev was holding on to a previous configuration. it started behaving like prod when I restarted the ingress controller pod. The issue was that the tls entries needed to match the same corresponding host entries under rules. So instead of specifying the wildcard domain as a server alias which works if you don't need https, I duplicated the host entries with the wildcard:

spec:
  tls:
  - secretName: dev-app-tls-cert
    hosts:
    - app.dev.dukketta.com
  - secretName: dev-wildcard-app-tls-cert
    hosts:
    - "*.app.dev.dukketta.com"    <---- this entry meets the entry below

  rules:
  - host: app.dev.dukketta.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dukketta-ui
          servicePort: 80
  - host: "*.app.dev.dukketta.com"   <---- needs this entry
    http:
      paths:
      - path: /
        backend:
          serviceName: dukketta-ui
          servicePort: 80

[rivernews]

This did solve the issue! Now other subdomain service are also using letsencypt and is cert secured. This looks a bit unintuitive though, since the http block info are redundant, the whole block is just for ingress to match the host in tls block. But this did solve the problem.