Kubernetes-ingress: Annotation to take priority in case of host conflict

[benjamin-bergia]

Is your feature request related to a problem? Please describe.
I have nginx configured as my ingress on my cluster with cert-manager and http validation.
cert-manager creates ingresses during the certification creation and validation process. These ingresses hosts consistently collide with the created virtualservers hosts.

Describe the solution you'd like
One simple solution would be to have an annotation that can be added to an ingress or VS to always take priority in case of host collision. After adding this annotation to the cert-manager ingress template, any temporary ingress created in the validation process would take priority over the application virtualserver. Once the certificate has been granted, cert-manager will remove the ingresses and the virtualserver will receive traffic again.

Describe alternatives you've considered
Looking at the way cert-manager generate its resources, I don't see any other simlle way. I would probably have to stop using VS and switch back to ingresses for all my apps, which at the moment is not possible.

Additional context
I do understand that it is not really an nginx ingress problem but it seems to be the simplest solution at the moment. Being able to create a new ingress or vs that will take priority over an existing one might also be useful during migrations or maintenance.

[anthonator]

I'm also interested in this specifically for the cert-manager use case. At the moment it doesn't seem possible to use cert-manager with virtualservers due to host collision.

[hostalp]

Did you try to use the acme.cert-manager.io/http01-edit-in-place: "true" annotation on the affected ingress resource? This solves the issue in my case, but it may not work with the virtualserver type.

The documentation for it is here https://cert-manager.io/docs/usage/ingress/#supported-annotations and it basically uses the existing ingress resource for the challenge instead of creating a new one (which would be conflicting with the existing one).

Another option is to use the nginx.org/mergeable-ingress-type: master annotation on the existing ingress resource and nginx.org/mergeable-ingress-type: minion on the temporary ones used to handle the challenge (https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/mergeable-ingress-types), but I'm not sure if the latter can be achieved kinda transparently (without having to set it manually once the temporary ingress resource is created by the cert-manager).

However what I'd like to achieve is to be able to configure the ingress controller to add the acme.cert-manager.io/http01-edit-in-place: "true" annotation to all ingress resources by default so I don't have to take care of each ingress resource individually. That would be nice to have.

Some additional ingress controller option to merge the conflicting ingress resource configs would be of course great too. Just a simple logic ensuring that the most specific match wins or so.

[benjamin-bergia]

@hostalp my understanding is that acme.cert-manager.io/http01-edit-in-place: "true" would work if my application was using an ingress . But since I am using virtualservers only it shouldn't work. This is currently impacting all my services and none of them is able to renew their certificates.
In my opinion, the controller should throw errors by default and refuse to load a new ingress / vs that if it uses a server_name already in use, unless mergeable-ingress-type is specified. That would make the issue clearer.

[030]

Cert-manager: v1.1.0

ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: boo
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - some-app.some-domain
    secretName: some-app.some-domain
  rules:
  - host: some-app.some-domain
    http:
      paths:
      - backend:
          serviceName: boo
          servicePort: 1234
        path: /

Addition of:

acme.cert-manager.io/http01-edit-in-place: "true"

did not work for us as well, as the cert remains invalid:

NAME                 READY            SECRET                  AGE
some-app.some-domain False       some-app.some-domain   74m

log:

I1216 19:07:00.219608       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="some-app.some-domain" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2jgnv" "related_resource_namespace"="keycloak" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="some-app.some-domain-bcn6v-1421178005-111683713" "resource_namespace"="keycloak" "resource_version"="v1" "type"="HTTP-01" 
I1216 19:07:00.219745       1 ingress.go:92] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="some-app.some-domain" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-x7fb8" "related_resource_namespace"="keycloak" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="some-app.some-domain-bcn6v-1421178005-111683713" "resource_namespace"="keycloak" "resource_version"="v1" "type"="HTTP-01" 
E1216 19:07:00.242763       1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="some-app.some-domain" "resource_kind"="Challenge" "resource_name"="some-app.some-domain-bcn6v-1421178005-111683713" "resource_namespace"="keycloak" "resource_version"="v1" "type"="HTTP-01"

Once the host is adjusted:

rules:
- host: some-app2.some-domain

the cert becomes valid.

Once the cert has been issued and the host is reverted to some-app.some-domain, the endpoint is valid.

How to ensure that the cert will be renewed automatically instead of changing the host manually every time?

[hostalp]

@030

Once the cert has been issued and the host is reverted to some-app.some-domain, the endpoint is valid.

And for which host is the new certificate issued? When you changed it to some-app2.some-domain it should be issued for this one so after going back to some-app.some-domain you should end up with a new one issued for your desired hostname.

One thing I noticed when adding the acme.cert-manager.io/http01-edit-in-place: "true" annotation was that I had to make sure a new certificate signing request is generated as the old one (generated when the config wasn't right) blocked any further operations for the same host. When you switched your config to other hostname and then back you may actually achieved that.
In short - if the new certificate is issued for some-app.some-domain (not for some-app2.some-domain) then you're likely fine.

By default the certificate is going to be renewed 30 days before the expiration of the current one so you'll have enough time to check whether it really works. Or create some test ingress and retry the same thing with that.

@benjamin-bergia
Yes I kind of expected it wouldn't work with the virtualserver resource. The Nginx docs mention that there's simply no merging support for virtualservers at all: https://docs.nginx.com/nginx-ingress-controller/configuration/handling-host-collisions/#merging-configuration-for-the-same-host
It looks like cert-manager should ideally support virtualservers to handle this correctly.

[benjamin-bergia]

@hostalp yeah agreed, but I don't think it's doable on cert-manager side to support all types of ingress resources under the sun. Maybe supporting templates for any kind of resources might be a solution on cert-manager side, in any case I would still be interested, on nginx side, by being able to define two resources (ingress or virtualserver) with the same host and manually set (via an annotation) which one is taking priority over the other. On top of solving the cert-manager case it will help with maintenance pages, and other situations where you temporarily want to enable/disable some services without rewriting your whole ingress.

@030 Unfortunately, I didn't find any solution at the moment. If you use gitops and all your conf is on a repo it makes it even worst since you can't manually switch domains like in your example.

[benjamin-bergia]

Actually, I don't think letting cert-manager modify my virtualServer or ingress in place is viable. I use argocd to manage my application. It means that any change to a resource that is not also changed in my repo will be rolled back. The only solution with gitops CD systems is for cert-manager to create a new, un-managed, ingress.

[030]

@hostalp The acme.cert-manager.io/http01-edit-in-place: "true" seems to solve the issue when a new ingress is deployed.

[benjamin-bergia]

I want to add a quick update on this.
In the past few days I have replaced most of my VSs with Mergeable ingresses. I also configured cert-manager to create ingresses as mergeable type minion. This currently doesn't work either. The reason is that when generating the certificate for the first time, nginx fails to re-load it's configuration since the master ingress reference the certificate (that doesn't exists yet) and the minion ingress needs the master to be present.

I will git a try to the edit-in-place annotation instead.

[benjamin-bergia]

So after some more work and few trial and errors, here is the only solution that allows to tie together nginx-ingress, cert-manager and argocd when using http01 challenge.

1. Configure cert-manager to create mergeable ingresses by adding the following to your issuer or clusterIssuer:

            ingressTemplate:
              metadata:
                annotations:
                  nginx.org/mergeable-ingress-type: minion

1. Create your ingresses with the following annotations:

annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: my-issuer
    nginx.org/mergeable-ingress-type: master

1. Move all your rules out of your master ingress and into your minion as mentioned in the doc.

A complete, minimum, example would be:

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: acme-staging
spec:
  acme:
    email: ***
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: acme-staging-key
    solvers:
      - http01:
          ingress:
            class: nginx
            serviceType: ClusterIP
            ingressTemplate:
              metadata:
                annotations:
                  nginx.org/mergeable-ingress-type: minion

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.org/mergeable-ingress-type: master
spec:
  tls:
    - hosts: [my.app.com]
      secretName: my-app-cert
  rules:
    - host: my.app.com

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app-frontend
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/mergeable-ingress-type: minion
spec:
  rules:
    - host: my.app.com
      http:
        paths:
            backend:
              serviceName: my-app
              servicePort: 80

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: my-app
  annotations:
    cert-manager.io/issue-temporary-certificate: 'true'
spec:
  secretName: my-app-cert
  dnsNames: [my.app.com]
  issuerRef:
    name: acme-staging
    kind: ClusterIssuer

[benjamin-bergia]

I realized that the temporary certificate annotation cannot be added to an ingress but a cert only. It means that I can't use the automated generation of certificates and had to create certificates for all my ingresses. I updated my previous example.