Boulder: Include SCT receipts in OCSP responses

[rolandshoemaker]

Blocked on submitting to CT logs (#95) and a lack of x509v3 extension support in crypto/ocsp. Once we have SCT receipts for certificate submissions to various logs we should include some subset of these receipts in the OSCP responses for certificates so they can actually be verified.

Depending on how many logs we choose to submit to we will also need a method that can choose some subset of the SCTs to include so that we don't bloat the OCSP response unnecessarily (perhaps one from a Google log + one from a non-Google log?)

[bifurcation]

x/crypto/ocsp now has support for extensions: https://go-review.googlesource.com/#/c/18202/

[hlandau]

This seems to now be blocked on https://github.com/cloudflare/cfssl/pull/488

[hlandau]

This is no longer blocked.

[JasonSome]

@rolandshoemaker @jsha So will this have priority now that the OCSP Extensions are enabled?

[hlandau]

May as well discuss the issues sooner rather than later.

So my plan is to deliver a byteslice to ca via RPC which is the verbatim body of the SCT OCSP extension, which ca includes when requesting OCSP signatures from the HSM. I think this is better than delivering arbitrary extensions via RPC because ca should be somewhat untrusting of its peers, and the security implications of being able to inject arbitrary SCT extension payloads (however that might happen) are surely fewer than those of being able to inject arbitrary OCSP extensions. Though I don't know how widely OCSP extensions are even used, so it may not matter that much.

That's the easy part. ISSUE ONE: The hard part is updating ocsp-updater so that it sends SCT data in its OCSP signing requests. Basically, ocsp-updater appears to contain several asynchronous elements, including OCSP updates and CT submission and storage of resultant SCTs. This means that when an OCSP response is generated, the availability of SCTs is not guaranteed.

So either ocsp-updater needs to be updated to express that dependency, perhaps by merging the CT submission and "initial OCSP response generation" systems into one lockstepped algorithm. This carries the hazard of tying the reliability of initial OCSP response generation to the reliability of CT logs. Use of different logs may ameliorate this liability. Thoughts on whether this is okay?

Another possibility would be to allow generation of OCSP responses without SCT staples, and then generate another set of OCSP responses triggered on receiving SCTs. This does increase the OCSP signing load, though, which I understand is the number one bottleneck for Let's Encrypt, so probably not desirable.

ISSUE TWO: I've amended the SQL queries used by ocsp-updater to use left joins to get all the sctReceipts as well as the certificateStatus in a single query. Thus if there are three sctReceipts the certificateStatus row is repeated three times, one for each sctReceipt row. If there are zero matching sctReceipts, the certificateStatus row is returned once with NULL values for the sctReceipt columns. This appears to confuse gorp. I've filed an issue: https://github.com/go-gorp/gorp/issues/309

[JasonSome]

@hlandau I would definitely agree with the "lockstepped" algorithm proposal, since the second proposal would effectively double the signing load, which apparently isn't desirable at all by Let's Encrypt.

[jmhodges]

One of the things to be worried about: we currently have something like 5 CT logs we send data to in production right now.

Do we want all of those CT log results stapled in? Do we just want one or some? If one or some, how do we configure that?

Are we okay with one CT log being down causing certificates to not be handed back to the user? How many CT logs would have to be down?

Note that the more remote services that are required to be up, the more chances to fail the request.

[rmhrisk]

That would kill your OCSP response size. I wouldnt do all. I would do minimum or include in certs vs OCSP.

[rolandshoemaker]

If I remember correctly serialized SCTs are generally <100 bytes (@rmhrisk you probably have better data on this?) so with two or three embedded we wouldn't be bloating responses... to much. Also with must-staple support this would be much more palatable.

[rolandshoemaker]

@hlandau a quick note on ocsp-updater, it does the initial OCSP generation, but boulder-ca initiates the first CT submission, ocsp-updater just searches for certificates without the expected number of SCT receipts. This means that most likely SCTs will be available when OCSP is initially generated, but it is still possible they won't be.

Myself, @jsha, @jmhodges, and other internal folks are going to discuss how we feel about this and will get back to you!

[hlandau]

@rolandshoemaker Precisely. The initial SCT submission is done by ca, but asynchronously and is in no way guaranteed.

One possibility is that you submit to all logs, and include the two SCTs which are returned fastest in OCSP. Might add a caveat on top of that that those two SCTs must be operated by different organizations (so one non-Google one). I await your conclusions.

[jsha]

Another possibility would be to allow generation of OCSP responses without SCT staples, and then generate another set of OCSP responses triggered on receiving SCTs. This does increase the OCSP signing load, though, which I understand is the number one bottleneck for Let's Encrypt, so probably not desirable.

The signing load is: 1 for the cert, 90 / (refresh period = 3 days) for the the OCSP responses. Presently that's 31 signatures per certificate. Adding another signature for the non-lockstepped case increases that to 32 signatures, a 3% increase. I think that's a worthwhile tradeoff to make sure we have an OCSP response ready as soon as possible.

In the medium term, we want to implement asynchronous issuance, where the client receives a URL to poll for the finished certificate. I think once we have that in place, the lockstepped approach will make a lot more sense. If it takes a few seconds or minutes to submit to a sufficient number of CT logs, we simply don't return the certificate to the client until that's done and we've incorporated the SCTs into the first signed OCSP response.

So my plan is to deliver a byteslice to ca via RPC which is the verbatim body of the SCT OCSP extension, which ca includes when requesting OCSP signatures from the HSM.

Agreed.

ISSUE TWO: I've amended the SQL queries used by ocsp-updater to use left joins to get all the sctReceipts as well as the certificateStatus in a single query. Thus if there are three sctReceipts the certificateStatus row is repeated three times, one for each sctReceipt row. If there are zero matching sctReceipts, the certificateStatus row is returned once with NULL values for the sctReceipt columns. This appears to confuse gorp. I've filed an issue: go-gorp/gorp#309

Thanks for filing the issue.

[eranmes]

A few comments:

• SCTs are indeed ~100 bytes each.
• Submission to CT logs can be done in parallel and pending submissions cancelled once "enough" SCTs have been obtained.
• Chrome's policy for EV certs requires one SCT from Google-operated log and one from non-Google operated log. There's no policy for DV certs yet but I think it's very unlikely if it'll be more strict than the policy for EV certs, so two SCTs should be plenty.
• CT logs should be pretty responsive (we're now gathering data on logs' latency, for Google's logs this should be under a second), so I should hope that would not delay certificate issuance too much.

Otherwise looking forward to this feature :)

[rolandshoemaker]

This has been depreciated by #2244.

[osirisinferi]

@jsha #2244 you're mentioning October as a "deadline" for SCT's in the certificate.. That's almost a whole year..

Isn't it possible SCT's through OCSP can be bridge between #2244 and nothing? Or would the deadline for SCT's through OCSP be about the same?

[jsha]

We're planning to implement SCTs in certificates well before the October deadline, though we don't have an exact launch date set yet. It doesn't make sense for us to implement SCTs in OCSP in the meantime since they would not be broadly enforceable (short of expect-ct pinning), and they would quickly become redundant.

[tijay]

@jsha hi, I'm wondering if the letsencrypt implemented (1) SCTs in certificates or (2) SCTs in OCSP-staple yet.
Thanks!

[tdelmas]

@tijay letsencrypt implemented (1) SCTs in certificates: https://letsencrypt.org/upcoming-features/

[jsha]

@tdelmas is correct, as usual. We're not planning to implement SCTs in OCSP anytime soon. Thanks!