Ingress-nginx: Referring to TLS secret from other namespace (i.e. not the namespace in which ingress is created)

[amit-kumar-4]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.): No

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): "Secret namespace"

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature

NGINX Ingress controller version: "0.10.2"

Kubernetes version (use kubectl version): v1.8.7

Environment:

• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): Ubuntu 16.04.3
• Kernel (e.g. uname -a): 4.11.0-1016-azure
• Install tools: Helm Chart v0.9.5 from https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/
• Others:

What happened:
When trying to use a TLS certificate using <namespace>/<secretName> pattern in tls section of ingress definition, Nginx controller still tries to get the details from the namespace where ingress was created.

What you expected to happen:
When TLS secret referred when creating an ingress is of pattern <namespace>/<secretName>, ingress controller shouldn't check only in ingress' namespace.

How to reproduce it (as minimally and precisely as possible):

1. Create 2 namespaces, say secret-store and ingress-store.
2. Create a secret containing a TLS certificate and key in secret-store namespace, say my-tls.
3. Create an ingress in ingress-store namespace with TLS enabled and in the .spec.tls.hosts[].secretName field put secret-store/my-tls to refer to the secret in secret-store namespace.
4. Check logs of ingress controller, line similar to below will be printed, indicating secret was never searched in secret-store namespace:
   W0305 11:39:26.826578 6 backend_ssl.go:49] error obtaining PEM from secret ingress-store/secret-store/my-tls: error retrieving secret ingress-store/secret-store/my-tls: secret ingress-store/secret-store/my-tls was not found

Anything else we need to know:
Initial glance on the code suggests that in below snippet, in the if block, we check if the secret has a / in it, and try to extract the secret from the namespace provided in ingress .spec.tls definition, it could work.

https://github.com/kubernetes/ingress-nginx/blob/164bb7bd57c684eeb21f123c2e85e474b8485162/internal/ingress/controller/controller.go#L1000-L1005

[mikebryant]

The standard security model for Kubernetes is that secrets are only accessible from within the current Namespace.

As per the API definition, the secretName field is purely a name reference, and cannot cross namespaces.

[amit-kumar-4]

True enough, but I think there is a catch-22 when trying to use wildcard certificate in cluster and keeping the certificate private key secure.
If I don't use the way mentioned above, I'll be exposing the private key for the wildcard certificate for anyone who has access to read secrets in the namespace, which will be many usually.

The standard security model for Kubernetes is that secrets are only accessible from within the current Namespace.

Since it will be the Ingress controller reading the secret, rather than something in the ingress namespace, I think your statement will still remain true.

[aledbf]

Closing. This should be allowed in the Ingress spec first. We are are not going to break this rule.

[sheerun]

@aledbf Is there a followup issue for changing the spec?

[aledbf]

@sheerun not yet

[mattalberts]

@aledbf @sheerun @aku105
During my evaluation of 0.14.0 and 0.15.0, I noticed that the ingress spec (as consumed by the nginx controller) would allow a mini-version of the issue described above without any patching (supported now).

• configure the controller with --default-ssl-certificate set to your wildcard cert
• define all ingress spec with hosts but do not specify the secretName

The ingress will be registered :80,:443 and use the default (wildcard certificate).

[Nowaker]

Thanks @mattalberts. While this is far from ideal (you can only have one default certificate), it's better than nothing.

That said, the spec should be updated to address the issue of using secrets from different namespaces. Wildcard certificates should be stored in the ingress controller's namespace, and Ingress resources from different namespaces should be allowed to reference them. What is the process of bringing up the spec discussion? Thanks! @sheerun @aledbf

[caleblloyd]

I look forward to an official fix coming in terms of a spec change that allows for referring to Secrets across namespace boundaries.

In the meantime, I am using a deployment that runs 2 kubectl containers - one to watch for new namespaces and copy the TLS Secret, and one to watch the TLS Secret for changes and apply to all namespaces. The code is here: ingress-cert-reflector.yml and I also wrote a corresponding blog post with detailed instructions.

[rayfoss]

UPDATE: in nginxinc/nginx-ingress default-ssl-certificate is default-server-tls-secret

---

@mattalberts I've tried to drop default-ssl-certificate in my controller, but I can't find where it goes

Dropping it in the spec gives ValidationError(DaemonSet.spec): unknown field "default-ssl-certificate" in io.k8s.api.extensions.v1beta1.DaemonSetSpec

my 1.2.0 controller spec

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  generation: 8
  labels:
    app: nginx-ingress
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx-ingress
    spec:
      containers:
      - args:
        - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
        - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
        - -v=3
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: nginx/nginx-ingress:1.2.0
        imagePullPolicy: IfNotPresent
        name: nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress
      serviceAccountName: nginx-ingress
      terminationGracePeriodSeconds: 30
  templateGeneration: 8
  updateStrategy:
    type: OnDelete

[aledbf]

Dropping it in the spec gives ValidationError(DaemonSet.spec): unknown field "default-ssl-certificate" in io.k8s.api.extensions.v1beta1.DaemonSetSpec

You are using a different ingress controller https://github.com/nginxinc/kubernetes-ingress where that flag is not supported

[abelal83]

@aledbf @sheerun @aku105
During my evaluation of 0.14.0 and 0.15.0, I noticed that the ingress spec (as consumed by the nginx controller) would allow a mini-version of the issue described above without any patching (supported now).

• configure the controller with --default-ssl-certificate set to your wildcard cert
• define all ingress spec with hosts but do not specify the secretName

The ingress will be registered :80,:443 and use the default (wildcard certificate).

@mattalberts could you show me an example ingress you have that this works with? I've setup as you suggested but when trying to access on https I get 404 but http works. Checking the nginx.conf file on nginx it only registers port 80.

[kfox1111]

Does your ingress that's 404ing for https have a tls section with the hostname in it but without secretName? the tls section needs to be there so it knows to bind the service to https too.

[abelal83]

That did the trick! Thank you.

[mohamedfarouk]

i tried the below work around and it worked for me

• Create your secret in the desired namespace

• Create dummy ingress in the same namespace, reference the created secret normally

it seems nginx load the ingress and associated secrets, and if there any other ingress in different namespace referenced the same shared secret by name it will be already loaded by the dummy ingress so nginx will use it.

[Nowaker]

@mohamedfarouk Wow. Thanks for sharing this.

[nanorobocop]

@mohamedfarouk I'm wondering how do you reference same shared secret by name from different namespace? Could you please share your configs?

[nanorobocop]

@mohamedfarouk ok, I've figured it out and it also worked for me with following config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dummy-load-certificates
  namespace: ingress-nginx
spec:
  tls:
  - hosts:
    - “*.example.com"
    secretName: cert-example.com
  rules:
  - host: “*.example.com"

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-app
  namespace: example-app
spec:
  tls:
  - hosts:
    - app.example.com
  rules:
    - host: app.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: example-app
            servicePort: 80

[janosi]

@aledbf would you consider an annotation like nginx.ingress.kubernetes.io/tls-secret-namespace acceptable to configure the namespace of the TLS secret in an Ingress?

[aledbf]

@janosi no, sorry. This is one of the rules defined in the ingress spec that we are not going to bend/break.

[zeeZ]

Note that the above workaround only works if the ingress containing the actual secret is the oldest one including that host. It will fall back to the default certificate and ignore all others. See also:

• #4926
• #2279

[janosi]

@aledbf I have hard times with finding the restriction in the Ingress specification https://kubernetes.io/docs/concepts/services-networking/ingress/
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#ingresstls-v1beta1-extensions
Maybe I look at the wrong specification? Thank you!

[aledbf]

@janosi Ingress predates the KEP process. For that reason, the are no formal documents besides the code. You can see the definitions and comments in types.go#L102. From the code, you can get two clear rules, service and secret are located in the same namespace. There is no way to reference a different one.
Also, Ingress exists before RBAC, something that introduces another dimension that needs to be considered, where there is no possibility to restrict access to one object.

These issues are being tackled in Ingress V2 https://github.com/kubernetes-sigs/service-apis where there is a clear delegation model and clear Roles and personas