Cert-manager: Operation cannot be fulfilled on certificates.cert-manager.io the object has been modified

[hubstairs-admin]

Hello,

I'm trying to issue certificate from a k8s Ingress, the cert is not becomes ready, the logs from cert-manager pod print this:

E1208 16:01:50.827738       1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"example-com-test-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="example/example-com-test-tls" 
E1208 16:01:50.872116       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"example-com-test-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="example/example-com-test-tls"

Here is the ingress part of my value.yaml :

ingress:
  enabled: true
  labels:
    traffic: "external"
    use-dns01-solver: "true"
  annotations: |
    kubernetes.io/ingress.class: haproxy
    cert-manager.io/cluster-issuer: letsencrypt-prod
    ingress.kubernetes.io/whitelist-source-range: "0.0.0.0"
    kubernetes.io/ingress.allow-http: "false"
    ingress.kubernetes.io/ssl-passthrough: "true"
  hosts:
    - host: example-test.hubstairs.com
      paths:
        - /
      serviceName: myapp
      servicePort: 3031

  tls:
    - secretName: example-com-test-tls
      hosts:
        - 'example-test.hubstairs.com'

The certificate has been successfully issue without error :

Events:
  Type    Reason     Age                  From          Message
  ----    ------     ----                 ----          -------
  Normal  Issuing    32s                  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  32s                  cert-manager  Stored new private key in temporary Secret resource "example-com-test-tls-6bt7d"
  Normal  Requested  32s                  cert-manager  Created new CertificateRequest resource "example-com-test-tls-xk85p"
  Normal  Issuing    12s                  cert-manager  The certificate has been successfully issued

When I try to access to url in a browser, I have an SSL error.

Environment details::

• Kubernetes version: 1.17
• Cloud-provider/provisioner: Google cloud
• cert-manager version: 1.1.0
• Install method: helm upgrade --install jetstack/cert-manager --version 1.1.0 -f value.yaml --debug

/kind bug

[alex-guenther]

I see the same errors in the log but everything works fine. You seem to have omitten the non error lines, which show that it eventually worked.

• Are sure the ClusterIssuer "letsencrypt-prod" is actually what the name implies?
• Did you check the certificatte, are the contents as expected?
• Could you resolve your issue or find more hints on what the issue is?

I would guess that this error log lines are not related to your actual problem. I opened a separate issue because I am not sure if we are dealing with the same problem. https://github.com/jetstack/cert-manager/issues/3516

[meyskens]

This error is a non-error for this, it is a known issue which we're looking into. I cannot seem to find the issue for it right now... it has to do with the design of the new controllers

[fliphess]

How can we work around it?

[hubstairs-admin]

@meyskens I going to update the HAProxy controller through the latest version. I'll keep you informed if the problem goes away.

[meyskens]

@fliphess cert-manager will work around these itself

[fliphess]

@meyskens

I did an upgrade to the latest cert-manager and all seems to be working, it's just that we notice this error with every (each and everyone) certificate request and renewal.

How should I look at this error? If cert-manager will work around these itself, should I consider it very verbose info output that causes some panic on my end or is this because of a bug or misconfiguration that is triggered?

If it's just very verbose info output about an issue that cert-manager will work around itself I'd love to be able to suppress it as it causes panic without reason. If it's not just extremely verbose info I'd love to have a fix so it doesn't appear anymore in my logs.

Edit: Forgot to ask my question: Can i fix the appearance of this "error" with configuration or is it something that should be fixed in the project itself?

[llb4ll]

Seeing the same error logs.

2020-12-22 14:24:24.150 CET
"cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="challenges.acme.cert-manager.io \"xxxxxxxx-2318237234-202910yyyyy\" not found" "key"="xxx/xxxxxxxx-pabcj-2318237234" "
Error
2020-12-22 14:24:24.138 CET
"cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"xxxxxxxx-2318237234-202910yyyyy\" not found" "
Error
2020-12-22 14:24:24.135 CET
"cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"xxxxxxxx-pabcj-2318237234-2792618003\" not found" "
Error
2020-12-22 14:24:24.117 CET
"cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"xxxxxxxx\": the object has been modified; please apply your changes to the latest version and try again" "key"="xxx/xxxxxxxx" "
Error
2020-12-22 14:24:24.105 CET
"cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="challenges.acme.cert-manager.io \"xxxxxxxx-pabcj-2318237234-3208871903\" not found" "key"="xxx/xxxxxxxx-pabcj-2318237234" "
Error
2020-12-22 14:24:24.105 CET
"cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"xxxxxxxx-pabcj-2318237234-3208871903\" not found" "
Error
2020-12-22 14:24:24.096 CET
"cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"xxxxxxxx-pabcj-2318237234-2243611777\" not found" "
Error
2020-12-22 14:24:24.088 CET
"cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"xxxxxxxx\": the object has been modified; please apply your changes to the latest version and try again" "key"="xxx/xxxxxxxx" "
Error
2020-12-22 14:24:24.083 CET
"cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"xxxxxxxx-pabcj-2318237234-3466251408\" not found" "
Error
2020-12-22 14:24:24.073 CET
"cert-manager/controller/CertificateIssuing "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"xxxxxxxx\": the object has been modified; please apply your changes to the latest version and try again" "key"="xxx/xxxxxxxx" "
Info
2020-12-22 14:24:24.016 CET
"Found status change for CertificateRequest "xxxxxxxx-pabcj" condition "Ready": "False" -> "True"; setting lastTransitionTime to XYZ"
Info
2020-12-22 14:24:24.015 CET
"cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="xxxxxxxx-pabcj-2318237234" "related_resource_namespace"="xxx" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="xxxxxxxx-pabcj" "resource_namespace"="xxx" "resource_version"="v1" "
Error
2020-12-22 14:24:23.124 CET
"ingress 'xxx/cm-acme-http-solver-ckdx2' in work queue no longer exists"

Environment details::

Kubernetes version: 1.16
Cloud-provider/provisioner: Google cloud
cert-manager version: 1.1.0
Install method: clean and re-install, using helm template generated files

[aogg]

cert-manager version: 1.1.0

install:
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--set ingressShim.defaultIssuerName=letsencrypt-prod \
--set ingressShim.defaultIssuerKind=ClusterIssuer \
--set ingressShim.defaultIssuerGroup=cert-manager.io \
--version v1.1.0

+1

[meyskens]

/priority important-soon
/area api

[nsteinmetz]

Hello @meyskens

So when we face this issue, we just should wait if I understand correctly ? any idea on how long I should wait ? Any workaround ?

Thanks !

[nsteinmetz]

Hello @meyskens

So when we face this issue, we just should wait if I understand correctly ? any idea on how long I should wait ? Any workaround ?

Thanks !

Ok my bad - as I cleaned a lot of annotations - I removed a useful one.

Once I redefined that certificate is to be generated with a ClusterIssuer, it works as expected...

[Patafix]

Same error here, but not working at all on my side...

I0203 09:15:05.917398 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "ssl-example.com-2whts" condition "Ready" to 2021-02-03 09:15:05.917387839 +0000 UTC m=+1221.665929458 I0203 09:15:05.917921 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "ssl-example.com-2whts" condition "Ready" to 2021-02-03 09:15:05.917914904 +0000 UTC m=+1221.666456523 I0203 09:15:05.918348 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "ssl-example.com-2whts" condition "Ready" to 2021-02-03 09:15:05.918341168 +0000 UTC m=+1221.666882787 I0203 09:15:05.919037 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "ssl-example.com-2whts" condition "Ready" to 2021-02-03 09:15:05.919030866 +0000 UTC m=+1221.667572495 I0203 09:15:05.919855 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "ssl-example.com-2whts" condition "Ready" to 2021-02-03 09:15:05.919849255 +0000 UTC m=+1221.668390874 E0203 09:15:05.954237 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-vault "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"ssl-example.com-2whts\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/ssl-example.com-2whts" E0203 09:15:06.117522 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"ssl-example.com-2whts\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/ssl-example.com-2whts" E0203 09:15:06.120400 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-venafi "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"ssl-example.com-2whts\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/ssl-example.com-2whts" E0203 09:15:06.121273 1 controller.go:158] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"ssl-example.com-2whts\": the object has been modified; please apply your changes to the latest version and try again" "key"="cert-manager/ssl-example.com-2whts"