Boulder: Recheck CAA when doing validation, even if the validation has not expired

[marktheunissen]

Hi team. Before we issue our multi-domain SAN certs, we check the validation for every domain on the cert to be issued. If they all succeed, we attempt to issue the cert, but the issuance can fail with CAA problems (SERVFAIL, timeout, prevents issuance). We then need to remove those domains that have bad CAA records and attempt to issue the cert without them.

To give you an idea of what kind of nasty hackery is required to do this, here are the regexes that parse the LE errors:

var dnsProbRegex = regexp.MustCompile(`(?i)^(?P<Domain>.*): DNS problem: (?P<Problem>.*) looking up CAA for (?P<CAARecordCheck>.*?)(,|\z)`)
var prevIssueRegex = regexp.MustCompile(`(?i)^(?P<Domain>.*): CAA record for (?P<CAARecordCheck>.*?) prevents issuance(,|\z)`)

These are applied to the slice of string that we first split out using split := strings.SplitAfter(strings.TrimSpace(msg), "While processing CAA for ")

Ideally, we would not be parsing Boulder errors in this brittle way. It would be preferable if the CAA recheck was done when requesting validation, even if the existing validation is still valid

[cpu]

Hi @marktheunissen

Before we issue our multi-domain SAN certs, we check the validation for every domain on the cert to be issued.

Could you add your own CAA check to this list of pre-validation steps for each domain? You should be able to replicate the SERVFAIL, timeout, etc responses using an Unbound instance configured with the ~same config we use in prod.

[marktheunissen]

Thanks @cpu, yeah we could do that. Is it not desirable to do the CAA recheck when getting a validation? Or is there some blocker preventing that from being done on Boulder's side?

[cpu]

Is it not desirable to do the CAA recheck when getting a validation? Or is there some blocker preventing that from being done on Boulder's side?

Maybe I'm not 100% sure what you're asking :-) Right now we do CAA checks at two points:

1. When a POST to a pending challenge is made and the validation authority attempts a TLS-SNI-01, DNS-01, or HTTP-01 challenge request.
2. When one of the names in the CSR POSTed to new-certificate has an authorization older than 8 hours.

It seems like you're asking for item 1 - that CAA be checked when validations are performed, but that's already the case. Maybe I'm misunderstanding and you're asking for CAA to also be rechecked when a valid authorization from a previous challenge validation is reused as the result returned from new-authz?

Unnecessary CAA lookups are somewhat undesirable because they increase the load on our systems (VA, DNS, Network, etc) without significant gains.

[marktheunissen]

asking for CAA to also be rechecked when a valid authorization from a previous challenge validation is reused as the result returned from new-authz?

Yes, I think that's right. :)

We want to follow a two step process:

Step 1: iterate over all domains on the list, and check each one has been validated by Let's Encrypt. We get a new auth and check it is valid.
Step 2: Immediately thereafter, issue the cert with confidence that there are no known problems with the domains on the CSR, that might result in LE returning an error.

Currently we can iterate over all the domains and get no errors, and then immediately attempt issuance only to get the CAA problem returned in the error message.

So, maybe we can we force the CAA recheck to happen in Step 1 by using the API slightly differently?

Or even better maybe do the 8 hour limit recheck even if the previous validation is reused?

[cpu]

Yes, I think that's right. :)

Great! Now we're on the same page :-)

So, maybe we can we force the CAA recheck to happen in Step 1 by using the API slightly differently?

You could deactivate your valid authorizations after you issue a certificate. Subsequent new-authz requests for the same name would return a fresh pending authorization (not the deactivated, previously valid authorization) and CAA checks would be performed.

Currently we can iterate over all the domains and get no errors, and then immediately attempt issuance only to get the CAA problem returned in the error message.

The issue of finding the domains that failed without having to do too much nasty error parsing is likely to improve in the "V2" API when we implement subproblems - The subproblems will include the identifier in question and won't require parsing to retrieve.

Or even better maybe do the 8 hour limit recheck even if the previous validation is reused?

Overall I think that since your goal is to only consume Let's Encrypt resources & attempt issuance when your own system is confident that the validation will succeed the most robust course of action is one in which you run an Unbound instance with a similar configuration to ours and test CAA/DNS lookups in your pre-flight checks. This will also help catch DNSSEC related errors ahead of time.

Between running an unbound instance, deactivating previously valid authorizations, and better error handling on the horizon I'm hesitant to write more code on our end & increase our DNS lookup volume. I think there are some good options here without making that change.

[marktheunissen]

Ok, thanks for taking the time to look at this, it was very helpful. I'll investigate the option of deactivating.

[cpu]

@marktheunissen Happy to help! Thanks for asking the question :-)

[jsha]

You could deactivate your valid authorizations after you issue a certificate. Subsequent new-authz requests for the same name would return a fresh pending authorization (not the deactivated, previously valid authorization) and CAA checks would be performed.

Note: Taking this path means higher validation load, and more opportunities for failures, depending on how often you reissue certificates. You may want to only deactivate authorizations after 8 hours so you get the maximum reuse potential out of them.

The issue of finding the domains that failed without having to do too much nasty error parsing is likely to improve in the "V2" API when we implement subproblems - The subproblems will include the identifier in question and won't require parsing to retrieve.

Note: Since subproblems are backwards compatible we can also implement them in the v1 API, which I hope to do so. @marktheunissen if you have the cycles to implement subproblems for the v1 API, we'd be happy to work with you on that and fix your regex problem. :-)

BTW, to get a better sense of use cases: how big are your multi-SAN certs, and what leads you to choose multi-SAN certs over single-SAN certs?

[marktheunissen]

I believe we do up to 100 SANs on a cert.

The reason for higher density is that we are charged per cert by our edge provider.

[cpu]

The reason for higher density is that we are charged per cert by our edge provider.

That's a shame :-( I thought the provider in question also made it difficult to rotate certificates in 90d intervals. Do you have to do that manually?