Cert-manager: Reuse / recovery of ExternalAccountBinding based account

[goblain]

Running against an ACME server that requires ExternalAcountBinding, after the initialization of the account and successful creation of private key secret associated with the account we take a backup of this secret for disaster recovery and reuse on other clusters. To validate if the backup process works I simple deleted and recreated CluesterIssuer leaving both eab secret and private key secret intact. The expectation was that as the secret was already there it should be used by cert-manager as is. Actual behaviour is that CM tries to reprocess the ExternaAccountBinding as newly created issuer has empty ACMEStatus, ultimately failing with

Failed to verify ACME account: 400 urn:ietf:params:acme:error:malformed: [External Account Binding] The account is not awaiting external account binding

As the binding is already processed and no longer available.

Environment details::

• Kubernetes version (e.g. v1.10.2): v1.15.5
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AKS
• cert-manager version (e.g. v0.4.0): v0.14.3
• Install method (e.g. helm or static manifests): helm

/kind bug

[elkh510]

hi
any updates ?

[munnerz]

What ACME server are you running against here? We will need to dig into the spec to see how ACME servers should behave if a call to the account endpoint submits an EAB payload along with it. I would have expected it just _ignores_ this data, but perhaps I'm wrong (there's a lot to the spec! 😅)

[munnerz]

Looking here: https://tools.ietf.org/html/rfc8555#page-40

   When a CA receives a newAccount request containing an
   "externalAccountBinding" field, it decides whether or not to verify
   the binding.  If the CA does not verify the binding, then it MUST NOT
   reflect the "externalAccountBinding" field in the resulting account
   object (if any).  To verify the account binding, the CA MUST take the
   following steps:

It reads like submitting this field should _not_ cause an error, and should instead simply 'drop' the field from the response.

[goblain]

In my case that is Sectigo ACME service. It seems to be pretty custom. Recently they even introduced their own challenges like sectigo-dns-01 which is painful enough for us to probably move away from using them in our kube/cert-manager based work as recent changes in our project enable us to make more use of wildcard certs then we had before (hence stay within LE quotas).

[arunramachandran15]

@munnerz @goblain, Any documentation on Sectigo CA intergation with cert-manager. All I could find in internet is Sectigo supports acme now. What should be the ClusterIssuer acme spec for Sectigo CA ?

[meyskens]

@arunramachandran15 we don't have access to a Sectigo ACME endpoint, any help or documentation is welcome!

[meyskens]

/triage needs-information

[mikesir87]

I believe this has been resolved with the merge of #3141, as that provides the ability to use a provided account rather than trying to create a new account.

[meyskens]

Closing this one. Feel free to /reopen if needed.

/close

[jetstack-bot]

@meyskens: Closing this issue.

In response to this:

Closing this one. Feel free to /reopen if needed.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[whereismyjetpack]

@meyskens I'm not sure If i'm using this correctly or not, but when I attempt to reuse a key minted from a previously valid registration the CRD never becomes ready with the following message

Failed to verify ACME account: 400 urn:ietf:params:acme:error:malformed: [External Account Binding] The account is not awaiting external account binding

it appears that it's using the key, but still trying to an EAB registration of some kind.

[dverbeek84]

Yes i have the same issue @whereismyjetpack, its still trying to doe the account binding.

[dverbeek84]

Looks like the register part is also used for verify.
For me here it is going wrong:

log: setup.go:218] cert-manager/controller/clusterissuers "msg"="failed to verify ACME account" "error"="400 urn:ietf:params:acme:error:malformed: [External Account Binding] The account is not awaiting external account binding" "related_resource_kind"="Secret" "related_resource_name"="sectigo-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="sectigo" "resource_namespace"="" "resource_version"="v1"

https://github.com/jetstack/cert-manager/blob/3177185a8938de181a76984f62ea97907342ec55/pkg/issuer/acme/setup.go#L213

I think we need to reopen the issue

/reopen

[jetstack-bot]

@dverbeek84: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[robstradling]

I'm lead developer for Sectigo's ACME service. I agree that the current behaviour (when a client attempts to recover an existing EAB-based account) can only be described as broken, but I don't currently see an RFC8555-compliant way to fix it. I've started a thread on the IETF ACME WG mailing list to discuss this issue (https://mailarchive.ietf.org/arch/msg/acme/WbAuLfHlRe-4jhmHcF0IoL1adSE/), but nobody has replied so far.

[btai24]

I have the same issue with zerossl + eab credentials. the credentials seem to only work once for spinning up a clusterissuer. if that clusterissuer would to be recreated using the same eab credentials, it never becomes ready.

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ErrVerifyACMEAccount 3m55s (x2 over 3m56s) cert-manager Failed to verify ACME account: 400 urn:ietf:params:acme:error:malformed: [External Account Binding] The account is not awaiting external account binding

[munnerz]

I don't think this is related to #3141 - and as noted above, cert-manager should handle this case better.

To reproduce this issue:

1) create an issuer that uses EAB (that is brand new/is not already registered)
2) wait for it to be registered
3) delete the issuer
4) recreate the issuer (utilising the same private key)

which causes the above mentioned event/warning to be posted onto the Issuer resource:

Warning  ErrVerifyACMEAccount  3m55s (x2 over 3m56s)  cert-manager  Failed to verify ACME account: 400 urn:ietf:params:acme:error:malformed: [External Account Binding] The account is not awaiting external account binding

@robstradling thanks for opening the question on the mailing list!

As noted here: https://github.com/jetstack/cert-manager/issues/2882#issuecomment-632768717 - personally my reading of those sections does _not_ indicate that you should reject the request if the account is already bound and the registration request already contains an externalAccountBinding field. Though it'd be great to get confirmation there 😄

In the shorter term, we _could_ consider switching the calls here:

https://github.com/jetstack/cert-manager/blob/7cef4582ec8e33ff2f3b8dcf15b3f293f6ef82cc/pkg/issuer/acme/setup.go#L326-L330

to instead _first_ call GetReg, and call RegisterAccount afterwards if the account does not exist. We once _did_ do it this way, however opted to call RegisterAccount first as it means only 1 request is made in the 'happy path' of a user creating a new Issuer, vs 2 calls being made if we switch it.

cc @jsha

[robstradling]

@munnerz There seems to be unanimous agreement that my interpretation was wrong! :-) @jsha's reply (https://mailarchive.ietf.org/arch/msg/acme/rwieiDldue3az6ZKb8plWvO9wFs/) to my list post explained the authors' intended meaning. I'm not totally happy that RFC8555 seems to use the word "reflect" inconsistently (https://mailarchive.ietf.org/arch/msg/acme/HrmrtOcW4sjc9HAIa3H3TATIB9w/), but, since nobody else seems to think that this is a problem, I'm now choosing to side with the majority opinion / authors' intent. So...
I've implemented a fix to Sectigo's ACME service that will permit recovery of EAB-based accounts. Due to holiday-related delays and planned deployment schedules, this fix is currently due to be deployed on January 10th 2021.
@btai24 acme.zerossl.com will also benefit from this fix.

[TafkaMax]

@munnerz There seems to be unanimous agreement that my interpretation was wrong! :-) @jsha's reply (https://mailarchive.ietf.org/arch/msg/acme/rwieiDldue3az6ZKb8plWvO9wFs/) to my list post explained the authors' intended meaning. I'm not totally happy that RFC8555 seems to use the word "reflect" inconsistently (https://mailarchive.ietf.org/arch/msg/acme/HrmrtOcW4sjc9HAIa3H3TATIB9w/), but, since nobody else seems to think that this is a problem, I'm now choosing to side with the majority opinion / authors' intent. So...
I've implemented a fix to Sectigo's ACME service that will permit recovery of EAB-based accounts. Due to holiday-related delays and planned deployment schedules, this fix is currently due to be deployed on January 10th 2021.
@btai24 acme.zerossl.com will also benefit from this fix.

I have the same problem with certbot, so this is related to Sectigos implementation of the ACME protocol, so if the fix is implemented it will also be applied to certbot or do I have to open a issue on their github aswell? (Also using Sectigo CA)

[robstradling]

I've implemented a fix to Sectigo's ACME service that will permit recovery of EAB-based accounts. Due to holiday-related delays and planned deployment schedules, this fix is currently due to be deployed on January 10th 2021.

This fix was deployed to Sectigo's ACME service on Jan 10th as planned.

@TafkaMax I would have expected any ACME client that supports recovery of EAB-based accounts to have benefited from this fix. If it's still not working for you, please open a ticket with our Support team.