Cert-manager: http01 is defeated by Ingress auth, cert-manager should have option to create own Ingress when using ingress-shim

[whereisaaron]

/kind feature

What happened:

Created an Ingress with these annotations:

  annotations:
    kubernetes.io/tls-acme: true
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
    kubernetes.io/ingress.class: nginx-prod
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: my-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"

The certificate was not issued because the ACME requests to /.well-known/acme/ received a 401 response.

172.24.10.56 - [172.24.10.56] - - [06/Jan/2018:23:35:27 +0000] "GET /.well-known/acme-challenge/NzRQOsG_eWiklofEKisjHprBHnPvXL1osIyz1SuKidc HTTP/1.1" 401 195 "-" "Go-http-client/1.1" 180 0.000 [foo-foo-operated-web] - - - -

What you expected to happen:

Certificate and Secret created. I expected the ingress-shim would create and Ingress for the /.well-known/acme/ path. And since that ingress has no auth, the ACME validation would succeed. I get this behavior with k-c-m because the nginx-ingress config scoops off and redirects all /.well-known/acme/ requests to k-c-m, before they hit the per-ingress config.

I think this problem comes up because cert-manager is modifying the existing Ingress (with auth) rather than creating an Ingress of it own for the /.well-known/acme/ path (without any auth).

How to reproduce it (as minimally and precisely as possible):

Add auth header to an Ingress as above.

Anything else we need to know?:

Environment:
kube-aws clusters on AWS.

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"clean", BuildDate:"2017-12-07T16:16:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"4292f9682595afddbb4f8b1483673449c74f9619", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

[euank]

I think this might be a little hairier due to the existence of many ingress controllers with various featuresets.

I know of at least one ingress controller which does not merge ingress rules with the same host (the nginxinc one).

How ingresses are merged is implementation dependent as well, so e.g. nginx might do the right tihng and apply auth to one, but not the other, but it would also be unsurprising if a similar ingress applied auth to both if it had merged them for having the same host.

I worry that the only way to handle this well with how ingress controllers currently behave is to do one of:

1. Have ingress specific logic for differences like this
2. Only support a ingress controllers which are common and behave well with whatever cert-manager chooses to do, even if some others don't
3. Entirely punt this to the user/ingress side of it, requiring user+ingress to be able to 'whitelist' the well-known path.

I think all of those options are somewhat reasonable. 2 seems like the best for a number of reasons, but if we're going with it, we should probably explicitly document the behaviour we expect and the ingress controllers we know work.

That being said, I don't entirely like changing behaviour of cert-manager, which should be generic, to work around an nginx-ingress-controller-specific quirk, so 3 seems a little more idealistically pretty.

I'm curious, could you make this work with 3 now by simply creating two ingress objects with the same host yourself, one with the auth annotations, and the other with the tls annotation and an arbitrary path/service which can be public (e.g. /.well-known/placeholder -> 'default-backend' or such?)

[munnerz]

It's also worth noting you can make cert-manager create a new ingress
resource instead of modifying an existing one by setting the 'ingressClass'
field instead of 'ingress' (see:
https://github.com/jetstack/cert-manager/blob/master/pkg/apis/certmanager/v1alpha1/types.go#L258)
on your Certificate resource.

This is not the default behaviour of the ingress shim however - the shim
itself does not actually solve acme challenges, instead it just creates
Certificate resources for any ingress resource it finds that specify the
appropriate annotations. We could expose some way to switch this behaviour,
either:

1. on a per ingress basis, with some kind of new annotation on the ingress
   resource
2. as a CLI flag specified to ingress-shim (so all or nothing, potentially
   more limiting in multi ingress controller clusters)
   On Sun, 7 Jan 2018 at 00:49, Euan Kemp [e-mail] wrote:

I think this might be a little hairier due to the existence of many
ingress controllers with various featuresets.

I know of at least one ingress controller which does not merge ingress
rules with the same host (the nginxinc one
https://github.com/nginxinc/kubernetes-ingress/blob/9eda8362/docs/nginx-ingress-controllers.md#the-key-differences
).

How ingresses are merged is implementation dependent as well, so e.g.
nginx might do the right tihng and apply auth to one, but not the other,
but it would also be unsurprising if a similar ingress applied auth to both
if it had merged them for having the same host.

I worry that the only way to handle this well with how ingress controllers
currently behave is to do one of:

1. Have ingress specific logic for differences like this
2. Only support a ingress controllers which are common and behave well
   with whatever cert-manager chooses to do, even if some others don't
3. Entirely punt this to the user/ingress side of it, requiring
   user+ingress to be able to 'whitelist' the well-known path.

I think all of those options are somewhat reasonable. 2 seems like the
best for a number of reasons, but if we're going with it, we should
probably explicitly document the behaviour we expect and the ingress
controllers we know work.

That being said, I don't entirely like changing behaviour of cert-manager,
which should be generic, to work around an
nginx-ingress-controller-specific quirk, so 3 seems a little more
idealistically pretty.

I'm curious, could you make this work with 3 now by simply creating two
ingress objects with the same host yourself, one with the auth headers, and
the other with the tls annotation and an arbitrary path/service which can
be public (e.g. /.well-known/placeholder -> 'default-backend' or such?)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/235#issuecomment-355790295,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMbP0c4rSdT73LM71U6RfEd5O64eMB2ks5tIBSCgaJpZM4RVfZR
.

[whereisaaron]

Thanks for the discussion @euank @munnerz.

Both strategies - new Ingress or patch the existing Ingress - seem clean and valid to me. I am certainly not suggesting one is wrong or always preferred.

@munnerz an Ingress annotation to choose which 'strategy' sounds excellent, with the user being able to choose which strategy (new Ingress or patch existing Ingress) best fits the behavior of their ingress controller, without cert-manager having to pay attention or care which ingress controller implementation is being used.

I agree the CLI flag would be a bit limiting if cert-manager aims to be a global cluster service, as multiple different ingress controller implementations are likely running in the same cluster for different application requirements.

@euank I think this is option '4' :-) Don't try to accommodate ingress controller foibles, punt that by letting the user control cert-manager's behavior.

@munnerz thanks for the tip about ingressClass vs ingress, that's already great control of the strategy used. I'll try it out.

[whereisaaron]

@munnerz that ingressClass / ingress strategy choice in Certificate resources is awesome! I realize that with that option I can issue certificates via http01 validation before even deploying the application or its Ingress. That's something I could only do with dns01 validation before.

An Ingress annotation equivalent to that strategy choice would still be very convenient. Especially for helm deployments, where most charts offer to inject Ingress annotations, but don't (yet :-) have an option to create cert-manager Certificate resources.

@euank @munnerz I can also confirm it works with authenticated apps under nginx-ingress, where you deploy the app with an authenticated Ingress first, then create the Certificate with ingressClass and http01. The authentication is correctly not applied to the Ingress+path that cert-manager creates, and the certificate is validated an issues, whilst the app remains authenticated. That's good news for renewals too.

So if the Ingress strategy annotation is added, you could deploy a typical helm chart, using its Ingress annotation setting, to deploy an authenticated HTTPS application.

[munnerz]

So it seems this issue specifically only effects people that do not want to manually configure their Certificate resources and instead use ingress-shim functionality to enable TLS on ingresses.

This again boils down to the issue of differences in ingress implementations. An annotation is a nice and easy escape hatch, and given the current default behaviour works on 'vanilla' ingresses for all ingress controllers (afaik), I think it'd be an acceptable addition.

We can pluck the ingress class value from the ingress itself, unless there's some use case for allowing a different class to be used? I'd rather not overcomplicate the API surface and make it harder to use if we can avoid it.

[whereisaaron]

True. Creating a Certificate resource is the way to do this with cert-manager. The annotation comes in a different section of the documentation under 'hacks we support to make it easy for people to adopt cert-manager' 😄

These annotations allow users to e.g. deploy a stable Helm chart without forking and modifying your own version to add Certificates. Or else organizing a separate chart or deployment for the Certificate.

[argusua]

Also, when using nginx-ingress-controller with enabled ssl-redirect, we got http response 308 for /.well-known/acme/ urls and renewing certificates functionality doesn't work.

[monotek]

Just run into exactly the same problem.
Imho this is a bug and not a feature.
In Kube-Lego this was not a problem.

[munnerz]

kube-lego managed to avoid this as you would explicitly configure the 'nginx ingress controller' as an option that worked cluster-wide. This means that we'd always create additional Ingress resources and not expose the choice (without changing CLI flags on kube-lego).

As mentioned before, an annotation to switch this behaviour would work well, but overloading annotations and pushing all this understanding onto end-users is painful.

What we need to decide to do something here:

• Do we set ingressClass or ingress by default?
  
  
  
  • Set ingress by default (what we do today): works for all kinds of ingress controller, including renewal, so long as additional annotations (e.g. auth) are not specified.
  
  
  • Set ingressClass by default: works only for nginx-like ingress controllers out of the box, and the user will then be forced to provide an additional annotation (eg certmanager.k8s.io/acme-http01-strategy: "in-place" or certmanager.k8s.io/edit-in-place) when using an ingress controller like GCE.
  
  

Whichever we choose, we need to ensure it is well documented. Having to understand so much about what is going on under the hood defeats the point of the abstractions we're trying to create, and eventually only serves to frustrate users (as they get to the point they understand the whole flow, and now they can't force cert-manager to do exactly what they want).

We need to also bare in mind that these annotations may be used for issuers other than ACME too (e.g. CA based issuer, and the upcoming Vault issuer). These issuers won't respect or have any need for this annotation, so this really is ACME specific.

Keen to hear thoughts on this, and I'm happy to integrate a solution.

[whereisaaron]

The clean abstraction only exists in an annotation-free environment (know any? 😄 ). As soon as we have any behavior annotations:

1. cert-manager has to be cloud-provider and/or ingress controller aware, before it can automatically do the right thing, and/or
2. cert-manager needs enough manual control over strategy and annotations, so that users who understand under-the-hood can tell cert-manager the correct strategy.

If we want to spare users the need to understand what they are doing, we have to build in cloud-provider and annotation-aware auto-magical behaviors, to see what cloud-provider, ingress controller, and annotation that user has decided to use, and pick the cert-manager strategy to navigate that.

And then the power users are going to want to be able to turn that auto-magical behavior off and specify exactly the strategy and annotations they want.

[whereisaaron]

In my mind, cert-manager creating and deleting its own Ingress resource for validation is always the better/clean default as opposed to modifying another application's Ingress. Where it can be avoided, I'd rather keep a clean separation between cert-manager and the applications' resources.

So I personally would default to creating an Ingress. I would include the kubernetes.io/ingress.class because is pretty much blessed now it has been promoted to the stable 'kubernetes.io' namespace. I would let power users add whatever additional annotations to the Ingress they think helpful.

The for the automatical behaviour, if cert-manager detects a problem environment, like GCE, or the nginxinc ingress controller (hard I guess?). It could log and switch to modifying Ingresses by default. I see these environments as edge cases, where the Ingress model doesn't work as intended.

Specifically specifying either ingressClass or ingress is going to lock the behavior and override the default. But we could try to detect and log when the user has painted themselves into a corner. It could be enough to give them a bunch of ideas or a help-page URL in the log message whenever the validation pre-check fail.

[whereisaaron]

Taking a step back from a single default. Maybe the helm chart could simply let you specify a cloud-provider and ingress controller, and then the chart could deploy cert-manager with the best defaults for that environment combo? Including even, adding some ingress-controller-specific annotations to the ClusterIssuer that the helm chart creates?

[munnerz]

Thanks for the details breakdown, I think you've summed up the situation well.

1. cert-manager has to be cloud-provider and/or ingress controller aware, before it can automatically do the right thing,

we've tried this with kube-lego, and it resulted in ingress classes as well as providers etc being implemented. This gets more and more complex as time goes on, and is something I really want to avoid building into cert-manager itself.

If we want to spare users the need to understand what they are doing, we have to build in cloud-provider and annotation-aware auto-magical behaviors, to see what cloud-provider, ingress controller, and annotation that user has decided to use, and pick the cert-manager strategy to navigate that.

Agreed

In my mind, cert-manager creating and deleting its own Ingress resource for validation is always the better/clean default as opposed to modifying another application's Ingress.

I agree with you, but I think the ship has sailed on this until future advancements to the upstream Ingress spec finally happen. It's a frustrating situation, but it is where it is and the issues will keep pouring in.

Where it can be avoided, I'd rather keep a clean separation between cert-manager and the applications' resources

:+1:

Taking a step back from a single default. Maybe the helm chart could simply let you specify a cloud-provider and ingress controller, and then the chart could deploy cert-manager with the best defaults for that environment combo? Including even, adding some ingress-controller-specific annotations to the ClusterIssuer that the helm chart creates?

This sounds ideal, and in a sense boils down to:

• Push forward with the toggle-able behavior we have now
• Better document example Issuer/ClusterIssuer & Certificate resources for various ingress controllers in our documentation
• Provide some form of automation for automatically setting up Issuers/ClusterIssuers/Certificates (this could be, as you say, in the form of a Helm chart).

FWIW, as discussed elsewhere, we can't really include our own CR resources inside the main helm chart at the moment so this would probably need to be implemented as an additional Helm chart. We could then also use this as a basis for generating static manifest examples for users too (using the existing static manifest generation script we use for the main chart)

[monotek]

I've already update my cluster to 1.9.2 which makes the situation a bit precarious as i can't go back to kube lego :-/ Imho this issue will become more urgent when more people trying to upgrade their clusters.

For now i had to remove my basic auth to get the cert and adding it again afterwards. But now i can't get automatic cert updates and removing the basic auth on production, even if its temporarily, isn't a real workaround.

Do you have some other idea how an automatic workaround could look like at the moment?

[munnerz]

@monotek see comments above:

It's also worth noting you can make cert-manager create a new ingress resource instead of modifying an existing one by setting the 'ingressClass' field instead of 'ingress' (see: https://github.com/jetstack/cert-manager/blob/master/pkg/apis/certmanager/v1alpha1/types.go#L258) on your Certificate resource.
This is not the default behaviour of the ingress shim however

By using the ingressClass field instead of the ingress field on the Certificate resource, you can mimic the behaviour of kube-lego when used with nginx.

[monotek]

Thanks for the hint.

I've added the "ingressClass: nginx" to my clusterissuer and i see that the well-known dir ist added as rule inside my kibana ingress. Nevertheless the basic auth still applies for the separate ingress rule and issuing the certificate still fails.

Kube-lego created an own inmgress for all well-known acme dirs outside of the ingress of the service i wanted to issue a certificate for.

How can i achieve that?

[munnerz]

You should be adding it as a field on you Certificate and not ClusterIssuer FYI. See here for an example: https://github.com/jetstack/cert-manager/blob/master/docs/examples/acme-cert.yaml#L22

With regards to basic auth still applying when a separate ingress is created, it seems like you are hitting this issue, which is a bug with ingress-nginx (which also applies to kube-lego too 😬): https://github.com/kubernetes/ingress-nginx/issues/2095

[monotek]

So i guess i can't use the helm chart (https://github.com/kubernetes/charts/tree/master/stable/cert-manager) at the moment if this is whats needed :-/

With kube-lego i did not hit the problem described in https://github.com/kubernetes/ingress-nginx/issues/2095

[munnerz]

I'm not sure what you mean - the Helm chart does not need changes in order to support this. You just need to create a Certificate resource that uses ingressClass instead of ingress. Please read the user guide on ACME HTTP01 for more info on how to configure this: https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/acme-http-validation.md

[monotek]

But the certificate resource is nothing you create manually when using the helm chart?

Eidt: I get it working when using:

• kubectl edit certificates kibana-tls -n default

After replacing "ingress: kibana" with "ingressClass: nginx" there the certificate is issued.

I would expect that i can set this option via the values.yaml file of the helm chart.

[munnerz]

What Helm chart are you using? Helm charts are not responsible for creating Certificate resources.

[monotek]

I use stable/cert-manager via:

• helm install stable/cert-manager --name cert-manager --namespace=default -f cert-manager-values.yaml

The values file looks like:

ingressShim:
  enabled: true
  # Optional additional arguments for ingress-shim
  extraArgs: ['--default-issuer-name=letsencrypt-staging', '--default-issuer-kind=ClusterIssuer']

createCustomResource: true

rbac:
  # Specifies whether RBAC resources should be created
  create: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true

My ingress is defined from the kibana helm chart:

• helm install stable/kibana --name kibana --namespace=default -f kibana-values.yaml

The values file for this looks like:

image:
  repository: "docker.elastic.co/kibana/kibana"
  tag: "5.4.2"
  pullPolicy: "IfNotPresent"

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/auth-realm: "Authentication required"
    nginx.ingress.kubernetes.io/auth-secret: kibana-auth
    nginx.ingress.kubernetes.io/auth-type: basic
    kubernetes.io/allow-http: 'false'
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
  hosts:
    - kibana.domain.com
  tls:
    - secretName: kibana-tls
      hosts:
        - kibana.domain.com

env:
  ELASTICSEARCH_URL: http://elasticsearch-elasticsearch-client:9200

rbac:
  create: true
  serviceAccountName: default

After installing both components i can edit the certificate via:

• kubectl edit certificates kibana-tls -n default

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  clusterName: ""
  creationTimestamp: 2018-02-23T12:20:46Z
  generation: 0
  name: kibana-tls
  namespace: infrastructure
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: kibana
    uid: f99e12a3-1893-11e8-a992-42010a9c0113
  resourceVersion: "25442"
  selfLink: /apis/certmanager.k8s.io/v1alpha1/namespaces/infrastructure/certificates/kibana-tls
  uid: f9a09d0e-1893-11e8-a992-42010a9c0113
spec:
  acme:
    config:
    - domains:
      - kibana.domain.com
      http01:
        ingress: "kibana"
  commonName: ""
  dnsNames:
  - kibana.domain.com
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  secretName: kibana-tls
status:
  acme:
    authorizations:
    - account: https://acme-staging.api.letsencrypt.org/acme/reg/5636959
      domain: kibana.domain.com
      uri: https://acme-staging.api.letsencrypt.org/acme/challenge/vrunjj9Sk0HDaKhAosEWxS3kc5h_zct93QdTYkePrdU/103867242
  conditions:
  - lastTransitionTime: 2018-02-23T12:23:08Z
    message: Certificate issued successfully
    reason: CertIssueSuccess
    status: "True"
    type: Ready

[munnerz]

There's not a change that needs to be made in the Helm chart for cert-manager is my point -

Instead, you need to opt to not use ingress-shim to automatically create a Certificate resource for you, and instead manually create the corresponding Certificate resource. The document below outlines all the details and nuances of this. Annotating an ingress resource is not the only way to trigger certificate issuance, and it doesn't offer the full flexibility that manually creating a Certificate resource does 😄. Take a look over the doc here, it's a page or two long but if you get through the whole thing I'd hope things are a bit clearer! https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/acme-http-validation.md

[monotek]

Wow... Together with https://github.com/jetstack/cert-manager/issues/289 the whole process just feels broken compared to the kube-lego helm chart😕

Imho there is a lot room for improvement here.

But thanks for your help anyway :-)

[whereisaaron]

@monotek I don't think that is true at all, rather you've suddenly a whole lot of choices and power that kube-lego never gave you. We do need good documentation and a good helm chart to make the migration and the learning curve for the new capabilities easy for people.

kube-lego only supports one issuer, one issuing protocol, one ingress controller, and one validation strategy. If that is all you need, and it works, then you're golden. If not, you're screwed. cert-manager supports multiple issuers (of multiple types), multiple protocols, multiple ingress controllers, and multiple validation strategies. Coming from kube-lego this is a bunch choices you just didn't even have before.

What I think is the problem is how to help migrating users so they don't have to think about these new choices if they are just replacing kube-lego. I think we need to improve with the helm chart to have a default install that emulates the kube-lego simplicity and makes those choices for you, defaulting to (only) way kube-lego works, e.g. creates a default ClusterIssuer for Let's Encrypt using 'http01'.

[tuananh]

defaulting to (only) way kube-lego works, e.g. creates a default ClusterIssuer for Let's Encrypt using 'http01'.

I think that would be awesome. There are probably many kube-lego users coming here.

[eseliger]

did you guys already decide on a way to handle this issue? would be down to help on the implementation as we really need this 🙂

[tankovich]

We're running into the ingress-shim issue as well. We can manually create the certificate resource with the ingressClass: nginx spec without issue, but would like to use the ingress-shim. Is there currently a way to implement this via adding ingress-shim annotation(s) to specify NGINX as the ingress controller (Helm deploy)?

[smelchior]

I am also running into this issue coming from kube-lego. Adding an option / annotation to make ingress-shim create the certificates with ingressClass: nginx would be awesome!

[LuckySB]

kubernetes dashboard with user authorithation need annotation
nginx.ingress.kubernetes.io/secure-backends: "true"
so acme request send to cert-pod via https...

i create certifcate by hand:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: k8dash-domain-tld
  namespace: kube-system
spec:
  secretName: k8dash-domain-tld
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  commonName: k8dash.domain.tld
  dnsNames:
  - k8dash.domain.tld
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - k8dash.domain.tld

[Justkant]

Hi @munnerz
Is someone working on adding support for ingress annotations that switches the ingress-shim behavior ?
So that we don't need to manually create/maintain Certificate Resources

[munnerz]

@Justkant not currently as far as I am aware. My time is completely pre-occupied trying to get ready for acmev2, Vault and the upcoming 0.3 release.

This would be a great first contribution from anyone looking to get involved 😄

[Justkant]

Ok, I try to look at what I can do.
I'm not a golang dev but we learn everyday by trying 😄

[Tim-Schwalbe]

Hi, I created a certificate with the ingress-shim and edited the certificate afterwards to ingressClass: nginx. Should this work and renew the certificate with basic-auth enabled?

[Jokero]

nginx-ingress 0.13.0 introduced no-auth-locations option with default value /.well-known/acme-challenge, so now it's possible and very easy to use nginx-ingress with basic auth + cert-manager with ingress-shim.
More details are here https://github.com/kubernetes/ingress-nginx/pull/2243

[monotek]

Imho this can be closed.
Works for me with nginx-ingress 0.13.0 image :-)

[Justkant]

IMHO We should wait for https://github.com/jetstack/cert-manager/pull/493 to be merge

[munnerz]

Closing as we've now got this option (and it is default in v0.3!) 🎉