Cert-manager: cannot issue certificated with cert manager, not sure why.

[DoryZi]

REQUEST FOR HELP?

/kind bug

/kind feature

I have installed the cert-manager, and it works. I have setup an issuer to use the http service per the documentation:
apiVersion: certmanager.k8s.io/v1alpha1

kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: default 
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    http01: {}

Then I tried to create a certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: staging-mydomain-ai
  namespace: default
spec:
  secretName: staging-mydomain-ai-tls
  issuerRef:
    name: letsencrypt-staging
  commonName: staging.mydomain.ai
  dnsNames:
  - staging.mydoamin.ai
  acme:
    config:
    - http01:
        ingress: mydomain-staging 
      domains:
      - staging.mydomain.ai

I have such an ingress working, but when I describe it seems to get stuck on getting the cert issued.
(I'm using google cloud platform) and the issuer is a namespace issuer (not cluster).
The kubectl is running in kube-system of course with the cert-manager, and my services are running in the same namespace (default)

rather than issue the certificate I get this:

ame:         staging-mydomain-ai
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"staging-mydomain-ai","namespace":"default"},"spec":...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Cluster Name:                   
  Creation Timestamp:             2018-01-27T20:01:10Z
  Deletion Grace Period Seconds:  <nil>
  Deletion Timestamp:             <nil>
  Generation:                     0
  Resource Version:               16691689
  Self Link:                      /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/staging-mydomain-ai
  UID:                            d1ef0cc4-039c-11e8-b8a7-42010a9c0017
Spec:
  Acme:
    Config:
      Domains:
        staging.mydomain.ai
      Http 01:
        Ingress:  mydomain-staging
  Common Name:    staging.mydomain.ai
  Dns Names:
    staging.mydomain.ai
  Issuer Ref:
    Name:       letsencrypt-prod
  Secret Name:  staging-mydomain-ai-tls
Events:
  Type     Reason                 Age   From                     Message
  ----     ------                 ----  ----                     -------
  Warning  ErrorCheckCertificate  3m    cert-manager-controller  Error checking existing TLS certificate: secret "staging-scoutr-ai-tls" not found
  Normal   PrepareCertificate     3m    cert-manager-controller  Preparing certificate with issuer
  Normal   PresentChallenge       3m    cert-manager-controller  Presenting http-01 challenge for domain staging.mydomain.ai
  Normal   SelfCheck              3m    cert-manager-controller  Performing self-check for domain staging.mydomain.ai

When I tried to re-run it generate a certificate, and when I moved to production again it got stuck like this. I don't know what's going on or why it cannot connect? or what it's stuck on the self perform self check? I don't have a clue what's going on.
Also how can I re-confgure the shim after to use ingress?
Can I use a local issuer or only clusterissuer?

[whereisaaron]

@DoryZi performing self-check means cert-manager is trying to make an HTTP request for http://staging.mydomain.ai/.wellknown/acme-challenge/... to make sure the challenge file is there before it asks the ACME server to check.

It will be stuck if it can't make that request because, e.g.

• the domain name staging.mydomain.ai doesn't exist
• the domain name staging.mydomain.ai doesn't resolve to the IP of your ingress controller
• your Ingress isn't working for some reason (e.g. target path annotation or forced ssl redirect)
• your network permission don't let cert-manager make the request

If it is something to do with your Ingress settings, like it uses rewrite-target see #286 for a way to deal with that.

[munnerz]

It's also worth noting, when using the GCP ingress controller updates to googles load balancers can take up to 10 minutes to propagate. cert-manager sets a timeout of 15 minutes on HTTP validations to allow for this.

Your configuration does look correct, so it'd be worth checking manually if it's possible to make the HTTP request yourself and get a 200 response. You can find the exact path you need to curl by inspecting your Ingress resource whilst cert-manager is performing the health check:

kubectl describe ingress mydomain-staging

[ericuldall]

When creating an ingress are we supposed to leave out the initial tls options?

For instance:

1. Create an ingress
2. Point mysite.com to ingress IP
3. Create cert-manager Certificate resource
4. Cert manager automagically adds tls to the targeted ingress ????! <---- not sure about this

Is that how the workflow should be setup?

[whereisaaron]

@ericuldall you either create a Certificate and Ingress, or an Ingress with the acme annotation. In both cases the Ingress should include the 'tls' reference to the certifiate/TLS Secret (whether that Secret exists yet or not).

The clean method it to create the Certificate resource first, using 'ingressClass' in the Certificate to identify your ingress controller. The cert-manager will create a temporary Ingress, issue the certificate and create the Secret. Then, whenever you do deploy the Ingress for your application, you don't need any Ingress annotation at all, and the TLS Secret will be there ready to go.

The alternative, convenience method is you deploy an Ingress with the acme annotation, and that refers to the TLS Secret (that doesn't exist yet). The ingress controller will just sit on the Ingress waiting for the named Secret to appear. cert-manager will see the Ingress annotation and create a Certificate resource for you, which will trigger the certificate to be issued and the certificate Secret created, When the Secret appears, the ingress controller will pick it up.

[ericuldall]

So referring back to the issue here...

If I create a certificate, how will the endpoint be verified when I don't yet have an IP to point my DNS at?

Since http verification will be done via http://mydomain.com/.wellknown/acme-challenge/... but I haven't setup A mydomain.com yet...since the Ingress hadn't existed yet. So then it fails 5 times and I have to wait an hour for my domain to get verified.

Is there some solution to this problem that I'm missing?

[munnerz]

Once the IP address is allocated, you should point mydomain.com at the
address, thus (eventually, once DNS TTLs expire/depending on your SOA
records) cert-manager will eventually be able to solve the challenge.

If you’d like to speed this process up, you can create your Ingress
resource first without the annotation, point your DNS, and then later add
the annotation.

Cert-manager is solely responsible for creating/modifying ingresss
resources for solving ACME challenges (or updating DNS TXT records if DNS
challenge verification is used), and storing those results in a Secret
resource. Whether that secret resource is used by an Ingress controller to
serve HTTP or not, is not cert-managers concern. The only requirement when
using HTTP validation is that the host name resolves to an IP address
associated with a particular Ingress (as denoted in the Certificate) or
pointed at a particular Ingress controller (as also could be denoted in the
Certificate via the ingressClass field).
On Mon, 29 Jan 2018 at 23:17, Eric Uldall [e-mail] wrote:

So referring back to the issue here...

If I create a certificate, how will the endpoint be verified when I don't
yet have an IP to point my DNS at?

Since http verification will be done via
http://mydomain.com/.wellknown/acme-challenge/... but I haven't setup A
mydomain.com yet...since the Ingress hadn't existed yet. So then it fails
5 times and I have to wait an hour for my domain to get verified.

Is there some solution to this problem that I'm missing?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/285#issuecomment-361420820,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMbPyhVQYTuH-e92aoEPnzsHk298143ks5tPlGYgaJpZM4Rvbbp
.

[munnerz]

If it’s not possible to know the IP for the Ingress beforehand or update
the annotations after the IP has been provisioned, you will need to update
your SOA records so that cert-manager checks DNS entries more regularly, so
that when you update the DNS zone, cert-manager uses the new IP sooner.
On Mon, 29 Jan 2018 at 23:25, James Munnelly [e-mail] wrote:

Once the IP address is allocated, you should point mydomain.com at the
address, thus (eventually, once DNS TTLs expire/depending on your SOA
records) cert-manager will eventually be able to solve the challenge.

If you’d like to speed this process up, you can create your Ingress
resource first without the annotation, point your DNS, and then later add
the annotation.

Cert-manager is solely responsible for creating/modifying ingresss
resources for solving ACME challenges (or updating DNS TXT records if DNS
challenge verification is used), and storing those results in a Secret
resource. Whether that secret resource is used by an Ingress controller to
serve HTTP or not, is not cert-managers concern. The only requirement when
using HTTP validation is that the host name resolves to an IP address
associated with a particular Ingress (as denoted in the Certificate) or
pointed at a particular Ingress controller (as also could be denoted in the
Certificate via the ingressClass field).
On Mon, 29 Jan 2018 at 23:17, Eric Uldall [e-mail]
wrote:

So referring back to the issue here...

If I create a certificate, how will the endpoint be verified when I don't
yet have an IP to point my DNS at?

Since http verification will be done via
http://mydomain.com/.wellknown/acme-challenge/... but I haven't setup A
mydomain.com yet...since the Ingress hadn't existed yet. So then it
fails 5 times and I have to wait an hour for my domain to get verified.

Is there some solution to this problem that I'm missing?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/285#issuecomment-361420820,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMbPyhVQYTuH-e92aoEPnzsHk298143ks5tPlGYgaJpZM4Rvbbp
.

[ericuldall]

Okay. I think it makes sense now, but just to be exhaustively clear...

First I will create this Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts:
    - www.mysite.com
    secretName: my-site-tls
  rules:
  - host: www.mysite.com
    http:
     paths:
     - backend:
         serviceName: my-site-service
         servicePort: 80

Once the IP address is allocated I will point the A record for www.mysite.com to the newly allocated address. Once I've confirmed www.mysite.com resolves to the Ingress I will update the ingress to match the following...

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts:
    - www.mysite.com
    secretName: my-site-tls
  rules:
  - host: www.mysite.com
    http:
     paths:
     - backend:
         serviceName: my-site-service
         servicePort: 80

At this point cert-manager will begin the process of verifying the endpoint and creating the tls secret. Does this sound like the correct way to go if I don't want to risk the failed verification timeout?

[munnerz]

Yep, exactly.

If you are concerned about these timeouts, I’d advise creating a second
issuer that uses the letsencrypt staging endpoint instead of production in
order to test this before moving to production 🙂
On Mon, 29 Jan 2018 at 23:32, Eric Uldall [e-mail] wrote:

Okay. I think it makes sense now, but just to be exhaustively clear...

First I will create this Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: "gce"
spec:
tls:

• hosts:
  
  
  
  • www.mysite.com
    
    secretName: my-site-tls
    
    rules:
  
  
• host: www.mysite.com
  http:
  paths:
  
  
  
  • backend:
    
    serviceName: my-site-service
    
    servicePort: 80
  
  

Once the IP address is allocated I will point the A record for
www.mysite.com to the newly allocated address. Once I've confirmed
www.mysite.com resolves to the Ingress I will update the ingress to match
the following...

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "gce"
spec:
tls:

• hosts:
  
  
  
  • www.mysite.com
    
    secretName: my-site-tls
    
    rules:
  
  
• host: www.mysite.com
  http:
  paths:
  
  
  
  • backend:
    
    serviceName: my-site-service
    
    servicePort: 80
  
  

At this point cert-manager will begin the process of verifying the
endpoint and creating the tls secret. Does this sound like the correct way
to go if I don't want to risk the failed verification timeout?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/285#issuecomment-361423956,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMbP44RT81AEiZOQK21toSJhkYcDIZ5ks5tPlUpgaJpZM4Rvbbp
.

[ericuldall]

I'm curious... How will it know which ClusterIssuer to use? Is there some other annotation we can put on the Ingress for that or is that what the Ingress shim is for?

[ericuldall]

I see there's the following:
certmanager.k8s.io/cluster-issuer is it required to use both that and the kubernetes.io/tls-acme: "true" annotation. Or can one or the other be used?

[munnerz]

Take a look at the documentation for ingress-shim for more info on its
behaviour. Not explicitly setting an Issuer will cause the shim to use the
defaults you specified whilst deploying.

https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/ingress-shim.md
On Tue, 30 Jan 2018 at 00:07, Eric Uldall [e-mail] wrote:

I see there's the following:
certmanager.k8s.io/cluster-issuer is it required to use both that and the
kubernetes.io/tls-acme: "true" annotation. Or can one or the other be
used?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/jetstack/cert-manager/issues/285#issuecomment-361430744,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMbPzOAjM-MqMPbN8a6j4THoTO2ZKBPks5tPl1YgaJpZM4Rvbbp
.

[Aleksion]

Hi,

I'm stuck with the same issue. I'm upgrading from Kube-Lego and followed the guide provided for that.
I'm fairly certain point nr. 1 and 2 (in @whereisaaron first response) isn't the issue.

point nr. 4: network permissions. How could I find out if that is the issue? and how would I got about solving it?

I tried running kube-lego again, to see if that one would work for me - in this case the health check also fails. It's trying to hit http://MYDOMAIN.COM/.well-known/acme-challenge/_selftest - which returns Page Not Found

Aren't cert-manager and/or kube lego supposed to manually create that challenge?

One thing that has changed is that I updated my cluster to 1.8 today. Would that affect anything?

UPDATE:
checking the logs of my main cluster, I can see multiple calls to /.well-known/acme-challenge/oRlUhcZxL11DQSi2dnC66nwI5mFs5vtKWJwKYsF4dJ with a response of 404 coming from my service.

UPDATE:
Tried using the approach with an ingressClass aswell. But how do I get it to hit the acme challenge if I haven't yet setup my ingress?

[whereisaaron]

For ingressClass Certificates, cert-manager creates its own temporary Ingress just of the challenge folder of the domain being verified. If requests are getting to your ingress controller and returning 404, sounds like you might need to check if your ingress controller is working right.

[munnerz]

@Aleksion it appears you've misconfigured DNS for MYDOMAIN.COM. You must ensure that this resolves to an address that is routed to your ingress controller else cert-manager (or kube-lego) will not be able to serve the challenge response, or the self check, at the appropriate path. For greater visibility, please open your own issue if you're unable to work through this.

I'm going to close this issue now as I think the original question has been resolved. @ericuldall feel free to re-open if you are still having issues. This ultimately seems like a documentation problem though.

/close

[ericuldall]

Got mine working okay ;)

[bbriggs]

It's really sad when this is the best documentation I can find on how to actually get cert-manager working.

[kragniz]

@bbriggs is there something in particular you'd like added to the docs?

[bbriggs]

I'm currently struggling through deploying this to GCP and will send a full list of gotchas when I'm done (hopefully in a PR), but basically there is no actual "you need these _N_ things to get it working." The "quick start" is "helm install stable/cert-manager" but in reality, it's "install cert-manager, install an issuer, and maybe install a certificate but we don't provide any clear path to doing that."

[FabioSchmidberger]

@bbriggs I am struggling to configure cert-manager on my GCP kubernetes cluster too. A guide or improved documentation would be awesome!

[Draiken]

Is this a specific issue with GCE Ingress since it creates the ingress with a new IP?

I've finally managed to get the certificate validated, but is this going to be needed every time a certificate needs to be renewed?

Previously I've had a much nicer experience since the ACME challenges were added to the existing ingress instead of creating a new one. Is this configurable?

[bbriggs]

@FabioSchmidberger, @Draiken thanks for the bump on this. I've had a pretty crappy summer but I'll work on it over the coming week and try to have something soon!

[Draiken]

I found the solution after carefully reading the docs.
By using the annotation certmanager.k8s.io/acme-http01-edit-in-place: "true" the old behavior can still be used.
I wasn't aware that the default behavior had changed, so I assumed this was specific to GCE.

I'll see if I add some documentation on this, since there's nothing here ATM: https://cert-manager.readthedocs.io/en/master/tutorials/acme/http-validation.html