Cert-manager: Operation cannot be fulfilled

Created on 17 Sep 2020 · 22Comments · Source: jetstack/cert-manager

Hello

Kubernetes version: 1.18.8

I have uninstalled cert-manager 0.15.2 completely and installed 1.0.1
After trying to issue certificate, the cert is not becomes ready, the logs from cert-manager pod print this:

E0917 15:07:19.412626       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"domain-co-il-wildcard-clusterissuer-tls-zskjr\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/domain-co-il-wildcard-clusterissuer-tls-zskjr" 
E0917 15:07:19.611836       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-ca "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"domain-co-il-wildcard-clusterissuer-tls-zskjr\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/domain-co-il-wildcard-clusterissuer-tls-zskjr" 
E0917 15:07:19.813780       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"domain-co-il-wildcard-clusterissuer-tls-zskjr\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/domain-co-il-wildcard-clusterissuer-tls-zskjr" 
E0917 15:07:20.021320       1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"domain-co-il-wildcard-clusterissuer-tls-zskjr\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/domain-co-il-wildcard-clusterissuer-tls-zskjr"

This is part of the events:

18m         Normal   Issuing             certificate/domain-co-il-wildcard-clusterissuer-tls                Issuing certificate as Secret does not contain a private key
18m         Normal   Generated           certificate/domain-co-il-wildcard-clusterissuer-tls                Stored new private key in temporary Secret resource "domain-co-il-wildcard-clusterissuer-tls-2f72v"
18m         Normal   Requested           certificate/domain-co-il-wildcard-clusterissuer-tls                Created new CertificateRequest resource "domain-co-il-wildcard-clusterissuer-tls-9xtgw"

This is the manifasts:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-wildcard-clusterissuer
  namespace: cert-manager
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: secret-issuer-account-key-nuc-25
    solvers:
      - dns01:
          cloudDNS:
            project: "xxxx-268209"
            serviceAccountSecretRef:
              name: clouddns-dns01-solver-svc-acct
              key: key.json
        selector:
          dnsNames:
            - 'domain.co.il'
            - '*.domain.co.il'
            - 'domain2.com'
            - '*.domain2.com'
````

apiVersion: v1
kind: Secret
metadata:
name: domain-co-il-wildcard-clusterissuer-tls
namespace: cert-manager
annotations:
kubed.appscode.com/sync: ""
type: kubernetes.io/tls
data:
ca.crt: ''
tls.crt: ''
tls.key: ''

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: domain-co-il-wildcard-clusterissuer-tls
namespace: cert-manager
spec:
commonName: ".domain.co.il"
secretName: domain-co-il-wildcard-clusterissuer-tls
renewBefore: 360h # 15d
dnsNames:
- domain.co.il
- ".domain.co.il"
issuerRef:
name: letsencrypt-wildcard-clusterissuer
kind: ClusterIssuer

```

Source

kfirfer

👀3

Most helpful comment

Same issue exists in V1.1.0

mazen423 on 7 Jan 2021

👍10

All 22 comments

Can you show us what certificaterequest domain-co-il-wildcard-clusterissuer-tls-9xtgw looks like, and then whatever order and challenge resources are referenced from there. (Installing https://www.postgresql.org/docs/9.1/queries-with.htm and then using kubectl cert-manager status certificate domain-co-il-wildcard-clusterissuer-tls would make this easier)

Also would you check letsdebug.net to see if that can tell you what is happening?

james-w on 17 Sep 2020

Name:         domain-co-il-wildcard-tls-gcfg8
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  cert-manager.io/certificate-name: domain-co-il-wildcard-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: domain-co-il-wildcard-tls-gfqlg
              meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2020-09-17T17:09:54Z
  Generate Name:       domain-co-il-wildcard-tls-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:generateName:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
        f:ownerReferences:
          .:
          k:{"uid":"4a53d33b-9668-493b-9342-cd42291d538e"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:request:
      f:status:
        .:
        f:conditions:
    Manager:    controller
    Operation:  Update
    Time:       2020-09-17T17:09:54Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  domain-co-il-wildcard-tls
    UID:                   4a53d33b-9668-493b-9342-cd42291d538e
  Resource Version:        9760355
  Self Link:               /apis/cert-manager.io/v1/namespaces/ingress-nginx/certificaterequests/domain-co-il-wildcard-tls-gcfg8
  UID:                     8d42d975-f0f1-4918-a990-31877ebd8e53
Spec:
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-wildcard-clusterissuer
  Request:  <base64>...
Status:
  Conditions:
    Last Transition Time:  2020-09-17T17:09:54Z
    Message:               Waiting on certificate issuance from order ingress-nginx/domain-co-il-wildcard-tls-gcfg8-3433298750: ""
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  2m42s  cert-manager  Created Order resource ingress-nginx/domain-co-il-wildcard-tls-gcfg8-3433298750
  Normal  OrderPending  2m42s  cert-manager  Waiting on certificate issuance from order ingress-nginx/domain-co-il-wildcard-tls-gcfg8-3433298750: ""

Name:         domain-co-il-wildcard-tls-gcfg8-3433298750
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  cert-manager.io/certificate-name: domain-co-il-wildcard-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: domain-co-il-wildcard-tls-gfqlg
              meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2020-09-17T17:09:54Z
  Generation:          1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          .:
          f:app.kubernetes.io/managed-by:
        f:ownerReferences:
          .:
          k:{"uid":"8d42d975-f0f1-4918-a990-31877ebd8e53"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:commonName:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:request:
      f:status:
    Manager:    controller
    Operation:  Update
    Time:       2020-09-17T17:09:54Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  domain-co-il-wildcard-tls-gcfg8
    UID:                   8d42d975-f0f1-4918-a990-31877ebd8e53
  Resource Version:        9760343
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/ingress-nginx/orders/domain-co-il-wildcard-tls-gcfg8-3433298750
  UID:                     52686b2f-a017-43d2-8429-05443da65cc4
Spec:
  Common Name:  *.domain.co.il
  Dns Names:
    *.domain.co.il
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-wildcard-clusterissuer
  Request:  <base64>
Events:     <none>

kfirfer on 17 Sep 2020

P.S.
When I use the annotation under ingress its worked
For example.: cert-manager.io/cluster-issuer: letsencrypt-wildcard-clusterissuer

But when I create manually certificate - it doesnt

Edit:
Not working, he worked for once but not more...

kfirfer on 17 Sep 2020

I think that this is another case of https://github.com/jetstack/cert-manager/issues/3267

james-w on 17 Sep 2020

after checking its really rate limit problem..
would be nice to see description in the logs that rate limit is reached , instead the current logs that dont describe the problem

kfirfer on 17 Sep 2020

Its seems not only rate limit problem..
I changed the domain to another one, and still same logs...

kfirfer on 17 Sep 2020

We are facing the same issue:

cert-manager-db79d5df4-lw4fr cert-manager E0918 09:29:52.061076 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"argocd-secret\": the object has been modified; please apply your changes to the latest version and try again" "key"="argocd/argocd-secret"

--- apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: email: "XXX" server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: letsencrypt-prod solvers: - selector: {} dns01: route53: region: eu-west-1

--- apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: argocd-server namespace: argocd annotations: cert-manager.io/cluster-issuer: letsencrypt-prod kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" nginx.ingress.kubernetes.io/ssl-passthrough: "true" spec: rules: - host: argocd.XXX http: paths: - backend: serviceName: argocd-server servicePort: https path: / tls: - hosts: - argocd.XXX secretName: argocd-secret

cert-manager-controller:v1.0.1
kubernetesVersion: 1.18.8

BartoszZawadzki on 18 Sep 2020

We are facing the same issues as above.

We currently have a working wildcard cert issued against one of our ingress in a different namespace:

NAMESPACE       NAME               READY   SECRET             AGE
vault-prod      letsencrypt-prod   True    letsencrypt-prod   23h
vault-restore   letsencrypt-prod   False   letsencrypt-prod   3m38s

The ingress is configured exactly the same as the working ingress but are getting the following within the cert-manager logs:

I0922 20:24:42.284233       1 conditions.go:173] Setting lastTransitionTime for Certificate "letsencrypt-prod" condition "Ready" to 2020-09-22 20:24:42.284228935 +0000 UTC m=+84038.307432839
I0922 20:24:42.284208       1 conditions.go:173] Setting lastTransitionTime for Certificate "letsencrypt-prod" condition "Issuing" to 2020-09-22 20:24:42.284195298 +0000 UTC m=+84038.307399199
E0922 20:24:42.294397       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"letsencrypt-prod\": the object has been modified; please apply your changes to the latest version and try again" "key"="vault-restore/letsencrypt-prod"
I0922 20:24:42.294556       1 conditions.go:173] Setting lastTransitionTime for Certificate "letsencrypt-prod" condition "Issuing" to 2020-09-22 20:24:42.294550562 +0000 UTC m=+84038.317754477
I0922 20:24:42.509186       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "letsencrypt-prod-8x498" condition "Ready" to 2020-09-22 20:24:42.509176441 +0000 UTC m=+84038.532380337

And looking at the order, it seems there were no events logged:

Name:         letsencrypt-prod-8x498-123322634
Namespace:    vault-restore
Labels:       app.kubernetes.io/instance=vault-restore
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=vault
              helm.sh/chart=vault-0.7.0
              traffic=internal
Annotations:  cert-manager.io/certificate-name: letsencrypt-prod
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: letsencrypt-prod-bc4qd
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2020-09-22T20:24:42Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  letsencrypt-prod-8x498
    UID:                   c88c7e99-8bd3-4374-a738-c8cf528c460e
  Resource Version:        7696849
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/vault-restore/orders/letsencrypt-prod-8x498-123322634
  UID:                     d8516d4e-20c2-4a74-93fa-79bb4c270d29
Spec:
  Dns Names:
    *.my.domain.io
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod
  Request:  <base64>
Events:     <none>

Also checking letsdebug.net shows that there are no issues with ratelimiting.

We've also redeployed cert-manager and its CRDs from scratch by deleting all existing cert-manager CRDs and deleting the cert-manager namespace. We also made sure that we have a ClusterIssuer resource to make sure we're using a cluster-wide resource to issue certificates.

I'm out of ideas on what the issue maybe, but it fails right after the certificate transitions into the Issuing... stage.

Meta:
Kubernetes Version: 1.17
cert-manager-controller: 1.0.0

kaypeter87 on 22 Sep 2020

@kaypeter87 did you also look at the Challenge resources?

meyskens on 24 Sep 2020

@kaypeter87 did you also look at the Challenge resources?

Yes, it seems none of the challenges for that order has been created.

kaypeter87 on 24 Sep 2020

Looks like sometimes (~0.5% of all certificates) cert-manager create unnecessary order on already renewed certificate.
And after it always come error cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"mydomain.com-fndl9\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/mydomain.com-fndl9"

It's all on fresh clean helm install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true without restore from backup (all resources recreated from ingress and secrets).
And its happens only on renewed certificates (and created from scratch certificates.cert-manager.io).
It's possible to get reason of this new orders? Currently I have 3 order like this (deleting Orders/Requests/Challenges didn't help), all certificates of this domains was renewed less 6 hours ago by this cert-manager. Other 1100+ certificates is OK.

TrurlMcByte on 26 Sep 2020

I appear to be seeing the same behavior as @kaypeter87 above, and thought that I had a workaround to offer.
Instead, this behavior quickly resurfaced and appears to be intermittent?

Due to the intermittent nature of this problem I am starting to suspect that it stems from rate-limiting, but can't offer any concrete evidence. letsdebug.net reports no problems with any subdomain + DNS-01, and I'm not sure how else to debug for rate-limiting.

As a side note: does letsdebug.net just always say "OK" no matter what?? I have yet to see it report a problem...

Verbose details can be found below

I have a fresh Kubernetes 1.19.2 cluster with a small-scale application running in the default namespace. I have also installed cert-manager in a namespace with the same name. I've configured an ACMEDNS + LetsEncrypt ClusterIssuer that appears to be working in the default namespace for issuing wildcard certs via LetsEncrypt DNS-01.

$ kubectl get cert
NAME               READY   SECRET             AGE
ndslabs-tls-auth   True    ndslabs-tls-auth   27h
ndslabs-tls-open   True    ndslabs-tls-open   27h
ndslabs-tls-root   True    ndslabs-tls-root   27h

In another namespace, I have a similar ingress rule set up that seems to be stuck:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: acmedns-issuer-prod
    kubernetes.io/ingress.class: nginx
    ndslabs.org/updated: 2020-09-27 04:34:10.235391921 +0000 UTC m=+96392.482403672
    nginx.ingress.kubernetes.io/auth-signin: https://www.hub.cheesehub.org/login/
    nginx.ingress.kubernetes.io/auth-url: https://www.hub.cheesehub.org/cauth/auth
  name: s2mvbc-sqlinjection-ingress
  namespace: sigite22
spec:
  rules:
  - host: s2mvbc-sqlinjection.hub.cheesehub.org
    http:
      paths:
      - backend:
          serviceName: s2mvbc-sqlinjection
          servicePort: 8888
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - '*.hub.cheesehub.org'
    secretName: s2mvbc-sqlinjection-tls

The Certificate / CertificateRequest are stuck with an Order that has no events and no Challenge resources are created:

$ kubectl get cert,certificaterequest,order,challenge -n sigite22
I0927 05:32:41.383477   23697 request.go:645] Throttling request took 1.110302164s, request: GET:https://192.168.0.4:6443/apis/autoscaling/v1?timeout=32s
I0927 05:32:51.383573   23697 request.go:645] Throttling request took 3.794775254s, request: GET:https://192.168.0.4:6443/apis/certificates.k8s.io/v1beta1?timeout=32s
I0927 05:33:01.388882   23697 request.go:645] Throttling request took 5.119265571s, request: GET:https://192.168.0.4:6443/apis/coordination.k8s.io/v1beta1?timeout=32s
I0927 05:33:11.389011   23697 request.go:645] Throttling request took 7.795506954s, request: GET:https://192.168.0.4:6443/apis/acme.cert-manager.io/v1alpha3?timeout=32s
NAME                                                  READY   SECRET                    AGE
certificate.cert-manager.io/s2mvbc-sqlinjection-tls   False   s2mvbc-sqlinjection-tls   59m

NAME                                                               READY   AGE
certificaterequest.cert-manager.io/s2mvbc-sqlinjection-tls-7w42b   False   59m

NAME                                                                  STATE   AGE
order.acme.cert-manager.io/s2mvbc-sqlinjection-tls-7w42b-2903943313           59m

No Challenge resource appears to be created, and the Order resource does not appear to have any events associated with it:

$ kubectl describe order -n sigite22
Name:         s2mvbc-sqlinjection-tls-7w42b-2903943313
Namespace:    sigite22
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: s2mvbc-sqlinjection-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: s2mvbc-sqlinjection-tls-r2c6m
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2020-09-27T04:34:10Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  s2mvbc-sqlinjection-tls-7w42b
    UID:                   87f7af68-f598-4085-9d62-8709bfbb66b4
  Resource Version:        406367
  Self Link:               /apis/acme.cert-manager.io/v1/namespaces/sigite22/orders/s2mvbc-sqlinjection-tls-7w42b-2903943313
  UID:                     6240089a-c5e8-433f-9135-c21bde6af1aa
Spec:
  Dns Names:
    *.hub.cheesehub.org
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   acmedns-issuer-prod
  Request:  <REDACTED>
Events:     <none>

As a test, I added the root domain alongside the wildcard in the TLS section and that initially appeared to fix things but the behavior came back:

  tls:
  - hosts:
    - hub.cheesehub.org
    - '*.hub.cheesehub.org'

bodom0015 on 27 Sep 2020

👍1

It appears my certificate having the issue has successfully been issued. I can safely say that it took 48 hours to issue and it appears to be a rate limiting problem. What worries me is that there were no indication of rate limiting in the debug logs and other new certificates were being issued during this problem. I cannot say, as there are many variables, if let's encrypt or cert-manager was failing. I do know that a certificate was failing to be issued when there were no indications of rate limiting.

kaypeter87 on 27 Sep 2020

👍1

@kaypeter87 surfacing of rate limit issues is improved in v1.1

meyskens on 29 Oct 2020

👍2

@meyskens I know that cert-manager v1.1 is on pre-release. I'm using the cert-manager's helm chart and the repo only has v1.0.4 available. Is this something that is coming to helm?

kaypeter87 on 19 Nov 2020

https://artifacthub.io/packages/helm/jetstack/cert-manager/1.1.0-alpha.1 1.1 is avaible @kaypeter87

anthr76 on 21 Nov 2020

v1.1.0 is released now

meyskens on 27 Nov 2020

🎉2

closing the issue since the new release is out, if the issue is reproduce I will create new issue

Thanks!

kfirfer on 28 Nov 2020

Same issue exists in V1.1.0

mazen423 on 7 Jan 2021

👍10

Same issue in v1.1.0 @kfirfer may you reopen please?

malacoss on 27 Jan 2021

Reopened

kfirfer on 27 Jan 2021

👍1

@malacoss would you open a new ticket with the information about the symptoms you are seeing. The code has changed and it will cause confusion to try and handle it here. Thanks.

james-w on 27 Jan 2021

Was this page helpful?

0 / 5 - 0 ratings