Cert-manager: Automatically re-issue certificate when issuerRef.name changes

[tikael1011]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
I was using https://acme-staging-v02.api.letsencrypt.org/directory for testing deployment. After I get
'Fake LE Intermediate X1' (I suppose I successfully deployed cert-manager, or migrated from kube-lego), I switched to https://acme-v02.api.letsencrypt.org/directory. Yet after cert-manager said 'successfully obtained certificate' and waited for a while, I did not see the new certificate.

Expected behaviour:
After I change to production acme server, I should get the 'true' certificate (https).

Steps to reproduce the bug:
Here is my cluster-issuer.yaml:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-private-key
  dns01:
    providers:
    - name: route53
      route53:
        region: us-east-2
        accessKeyID: MYKEY
        hostedZoneID: MYZONE
        secretAccessKeySecretRef:
          name: MYSECRET
          key: secret-access-key

Here is my Certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  generation: 1
  name: MYCERT
  namespace: pre
spec:
  acme:
    config:
    - dns01:
        provider: route53
      domains:
      - my.xx.com
      - '*.my.xx.com'
  commonName: ""
  dnsNames:
  - my.xx.com
  - '*.my.xx.com'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  secretName: letsencrypt-private-key

Anything else we need to know?:
route53 iam settings:

Action Resource  Request condition
List (2 of 16 actions)
GetChange ChangeID | string like | All None
ListHostedZonesByName All resources None
Write (1 of 23 actions)
ChangeResourceRecordSets HostedZoneID | string like | All None

some logs from cert-manager:

I0810 00:15:16.184621       1 sync.go:259] Preparing certificate MYCERT with issuer
I0810 00:15:16.184679       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 00:15:16.185124       1 logger.go:27] Calling GetOrder
I0810 00:15:16.413745       1 logger.go:52] Calling GetAuthorization
I0810 00:15:16.498555       1 logger.go:52] Calling GetAuthorization
I0810 00:15:16.588827       1 prepare.go:263] Cleaning up old/expired challenges for Certificate MYCERT
I0810 00:15:16.588868       1 helpers.go:188] Found status change for Certificate "MYCERT" condition "ValidateFailed": "False" -> "False"; setting lastTransitionTime to 2018-08-10 00:15:16.58886399 +0000 UTC m=+18876.383905566
I0810 00:15:16.588942       1 sync.go:266] Issuing certificate...
I0810 00:15:16.589143       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 00:15:16.589947       1 logger.go:27] Calling GetOrder
I0810 00:15:16.789966       1 logger.go:37] Calling FinalizeOrder
I0810 00:15:17.259308       1 sync.go:273] Error issuing certificate for MYCERT: error getting certificate from acme server: acme: urn:ietf:params:acme:error:malformed: Order's status ("valid") is not acceptable for finalization
I0810 00:15:17.259996       1 sync.go:191] Certificate MYCERT scheduled for renewal in 1438 hours
E0810 00:15:17.266799       1 controller.go:186] certificates controller: Re-queuing item "MYCERT" due to error processing: error getting certificate from acme server: acme: urn:ietf:params:acme:error:malformed: Order's status ("valid") is not acceptable for finalization
I0810 00:16:17.267135       1 controller.go:177] certificates controller: syncing item 'MYCERT'
I0810 00:16:17.267899       1 sync.go:259] Preparing certificate MYCERT with issuer
I0810 00:16:17.267933       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 00:16:17.268351       1 prepare.go:247] Cleaning up previous order for certificate MYCERT
I0810 00:16:17.268398       1 prepare.go:263] Cleaning up old/expired challenges for Certificate MYCERT
I0810 00:16:17.268435       1 logger.go:22] Calling CreateOrder
I0810 00:16:17.702497       1 acme.go:196] Created order for domains: [{dns *.my.xx.com} {dns my.xx.com}]
I0810 00:16:17.702629       1 logger.go:52] Calling GetAuthorization
I0810 00:16:17.765005       1 logger.go:52] Calling GetAuthorization
I0810 00:16:17.896835       1 prepare.go:263] Cleaning up old/expired challenges for Certificate MYCERT
I0810 00:16:17.896903       1 helpers.go:188] Found status change for Certificate "MYCERT" condition "ValidateFailed": "False" -> "False"; setting lastTransitionTime to 2018-08-10 00:16:17.896888312 +0000 UTC m=+18937.691929886
I0810 00:16:17.896952       1 sync.go:266] Issuing certificate...
I0810 00:16:17.897030       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 00:16:17.897758       1 logger.go:27] Calling GetOrder
I0810 00:16:18.008889       1 logger.go:37] Calling FinalizeOrder
I0810 00:16:19.253176       1 issue.go:104] successfully obtained certificate: cn="my.xx.com" altNames=[my.xx.com *.my.xx.com] url="https://acme-v02.api.letsencrypt.org/acme/order/123456/7890"
I0810 00:16:19.262023       1 sync.go:285] Certificate issued successfully
I0810 00:16:19.263186       1 helpers.go:188] Found status change for Certificate "MYCERT" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2018-08-10 00:16:19.263174008 +0000 UTC m=+18939.058215550
I0810 00:16:19.263821       1 sync.go:191] Certificate MYCERT scheduled for renewal in 1438 hours
I0810 00:16:19.269514       1 controller.go:191] certificates controller: Finished processing work item "MYCERT"
I0810 00:17:19.268936       1 controller.go:177] certificates controller: syncing item 'MYCERT'
I0810 00:17:19.269762       1 sync.go:191] Certificate MYCERT scheduled for renewal in 1438 hours
I0810 00:17:19.269875       1 controller.go:191] certificates controller: Finished processing work item "MYCERT"

Event log from describe MYCERT:

Status:
  Acme:
    Order:
      URL:  https://acme-v02.api.letsencrypt.org/acme/order/123456/7890
  Conditions:
    Last Transition Time:  2018-08-10T00:16:17Z
    Message:               Order validated
    Reason:                OrderValidated
    Status:                False
    Type:                  ValidateFailed
    Last Transition Time:  2018-08-10T00:16:19Z
    Message:               Certificate issued successfully
    Reason:                CertIssued
    Status:                True
    Type:                  Ready
Events:
  Type     Reason          Age                From          Message
  ----     ------          ----               ----          -------
  Warning  IssuerNotReady  38m (x5 over 40m)  cert-manager  Issuer letsencrypt-staging not ready
  Warning  IssueError      37m                cert-manager  Failed to finalize order: acme: urn:ietf:params:acme:error:malformed: Order's status ("valid") is not acceptable for finalization
  Normal   CreateOrder     36m (x2 over 1h)   cert-manager  Created new ACME order, attempting validation...
  Normal   IssueCert       36m (x3 over 1h)   cert-manager  Issuing certificate...
  Normal   CertObtained    36m (x2 over 1h)   cert-manager  Obtained certificate from ACME server
  Normal   CertIssued      36m (x2 over 1h)   cert-manager  Certificate issued successfully

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.10.5-gke.3
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.4.0
• Install method (e.g. helm or static manifests): helm

/kind bug

[munnerz]

So right now, cert-manager does not invalidate a certificate based on the issuer changing. This is something we should support however, as there's also not an easy way to force a new certificate to be issued (aside from deleting the target secret, which would cause downtime for anything utilise that cert!).

Can you confirm that deleting the secret named in certificate.spec.secretName alleviates the issue?

[tikael1011]

@munnerz do you mean apply the following yaml?

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  generation: 1
  name: MYCERT
  namespace: pre
spec:
  acme:
    config:
    - dns01:
        provider: route53
      domains:
      - my.xx.com
      - '*.my.xx.com'
  commonName: ""
  dnsNames:
  - my.xx.com
  - '*.my.xx.com'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging

[munnerz]

Hm - just noticed you have certificate.spec.secretName set to letsencrypt-private-key - this should be set to the name of the secret that you want to contain your actual signed TLS cert. The Let's Encrypt private key is different, and must be stored separately.

But what I meant, is running kubectl delete secret {NAME OF THE SECRET NAMED ON THE CERTIFICATE HERE}, which cert-manager will then notice and automatically trigger a re-issue when it sees the secret missing.

[tikael1011]

I did as you suggested, deleted the secret and apply -f

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  generation: 5
  name: MYCERT
  namespace: pre
spec:
  acme:
    config:
    - dns01:
        provider: route53
      domains:
      - my.xx.com
      - '*.my.xx.com'
  commonName: ""
  dnsNames:
  - my.xx.com
  - '*.my.xx.com'
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  secretName: MY_CERT_NAME

and got the following cert manager log:

1 sync.go:182] [MY_CERT_NAME] Error getting certificate 'rh-certs': secret "rh-certs" not found
E0810 09:41:10.448308       1 controller.go:186] certificates controller: Re-queuing item "MY_CERT_NAME" due to error processing: error getting certificate from acme server: acme: urn:ietf:params:acme:error:malformed: Order's status ("valid") is not acceptable for finalization
I0810 09:41:14.448530       1 controller.go:177] certificates controller: syncing item 'MY_CERT_NAME'
I0810 09:41:14.449192       1 sync.go:259] Preparing certificate MY_CERT_NAME with issuer
I0810 09:41:14.449426       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 09:41:14.450183       1 prepare.go:247] Cleaning up previous order for certificate MY_CERT_NAME
I0810 09:41:14.450353       1 prepare.go:263] Cleaning up old/expired challenges for Certificate MY_CERT_NAME
I0810 09:41:14.450580       1 logger.go:22] Calling CreateOrder
I0810 09:41:14.841852       1 acme.go:196] Created order for domains: [{dns *.my.xx.com} {dns my.xx.com}]
I0810 09:41:14.842125       1 logger.go:52] Calling GetAuthorization
I0810 09:41:14.912237       1 logger.go:52] Calling GetAuthorization
I0810 09:41:14.971793       1 prepare.go:263] Cleaning up old/expired challenges for Certificate MY_CERT_NAME
I0810 09:41:14.972137       1 helpers.go:188] Found status change for Certificate "rh-certs" condition "ValidateFailed": "False" -> "False"; setting lastTransitionTime to 2018-08-10 09:41:14.972129881 +0000 UTC m=+52834.767171453
I0810 09:41:14.972520       1 sync.go:266] Issuing certificate...
I0810 09:41:14.973343       1 acme.go:162] getting private key (letsencrypt-private-key->tls.key) for acme issuer kube-system/letsencrypt-staging
I0810 09:41:14.973912       1 logger.go:27] Calling GetOrder
I0810 09:41:15.258540       1 logger.go:37] Calling FinalizeOrder
I0810 09:41:15.982940       1 issue.go:104] successfully obtained certificate: cn="my.xx.com" altNames=[my.xx.com *.my.xx.com] url="https://acme-v02.api.letsencrypt.org/acme/order/123/456"
I0810 09:41:16.015110       1 sync.go:285] Certificate issued successfully
I0810 09:41:16.016603       1 helpers.go:188] Found status change for Certificate "rh-certs" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2018-08-10 09:41:16.016589222 +0000 UTC m=+52835.811630786
I0810 09:41:16.018545       1 sync.go:191] Certificate MY_CERT_NAME scheduled for renewal in 1438 hours
I0810 09:41:16.024177       1 controller.go:191] certificates controller: Finished processing work item "MY_CERT_NAME"
I0810 09:41:18.024620       1 controller.go:177] certificates controller: syncing item 'MY_CERT_NAME'
I0810 09:41:18.025370       1 sync.go:191] Certificate MY_CERT_NAME scheduled for renewal in 1438 hours
I0810 09:41:18.025432       1 controller.go:191] certificates controller: Finished processing work item "MY_CERT_NAME"

Does this imply I should be fine?

[munnerz]

Yep that looks like it issued correctly.

This message: E0810 09:41:10.448308 1 controller.go:186] certificates controller: Re-queuing item "MY_CERT_NAME" due to error processing: error getting certificate from acme server: acme: urn:ietf:params:acme:error:malformed: Order's status ("valid") is not acceptable for finalization

is fixed in the v0.4.1 release (which is due to be tagged/released today).

I'm going to rename this issue, as ultimately this is a feature request 😄 we should detect that a certificates issue has changed, and force a re-issue automatically in future!

[tikael1011]

Yes, problem solved. I think you pointed one important thing, which is not mentioned/ambiguous in the kube-lego -> cert-manager migration guide (or maybe the guide is fine but I understood wrong): when manually issue certificate, use own secret, apart from letsencrypt-private-key

[munnerz]

@tikael1011 if there's any suggestions for improvements, you should be able to open a PR all through your browser by just editing this doc: https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/acme/migrating-from-kube-lego.rst 🙏

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[k]

@munnerz can confirm when changing issuer in a Certificate from staging to prod it does not create a new certificate unless the old one is deleted. It sounds like this is expected behavior, but most of the time the process involves first testing the certificate acquisition by using staging and then changing the issuer to prod. Maybe instead of deleting the certificate cert-manager could edit the existing certificate to the newly acquired one from the issuer?

[chrissound]

Is there a temporary solution / work around until this is fixed?

[tikael1011]

@chrissound you can 'kubectl delete secret {NAME OF THE SECRET NAMED ON THE CERTIFICATE HERE}' and cert-manager should be able to re-issue.

[chrissound]

This does not seem to update the ingress however? @tikael1011 I suppose that would be the issue with the ingress controller itself though... Perhaps it takes a while.