Boulder: Delay between Certificate Issuance and Availability of OCSP

[msanjay94]

Hi,
Recently, I'm getting error code "6" in OCSP requests for certificates purchased from Let's Encrypt. The OCSP query will be triggered automatically immediately after certificate purchase. Upon retrying the OCSP request after a while, the response is obtained successfully.

I posted a question regarding this in the forums, and I was asked to report a bug with details on the issue.

The details are as follows,

Serial Number of Certificate : 04:60:b9:ec:1c:d1:05:d5:ec:ea:a2:be:ff:95:41:11:76:83
Time of purchase : Tue Nov 21 2017 01:35:49 UTC
Time of OCSP Query : Tue Nov 21 2017 01:36:18 UTC

Response Code from OCSP Responder : 6

Both ACME and OCSP clients are my own implementations. Let me know if i should provide further information regarding the issue.

Thanks,
Sanjaykumar M

[jsha]

To rule out a bug in your OCSP client, can you please try to reproduce this result using https://certificate.revocationcheck.com/ ? Thanks!

[17dec]

I've been noticing the same behavior recently. I'm using the Dehydrated client (https://github.com/lukas2511/dehydrated) to create the certificates, and then follow up with an OCSP request using openssl:

  openssl ocsp -noverify -no_nonce -issuer chain.pem -cert cert.pem -VAfile chain.pem \
      -url http://ocsp.int-x3.letsencrypt.org -header Host ocsp.int-x3.letsencrypt.org \
      -respout ocsp.der~

This usually works, but sometimes results in a "Responder Error: unauthorized (6)". I am unable to get a positive response from the OCSP server for a while for the same certificate when this happens, which might suggest that the negative response remains in a cache somewhere (I am not using a proxy myself, though). The certificate still validates just fine with https://certificate.revocationcheck.com/.

(FWIW, I'm quering the OCSP server from France).

In case it helps, here's a certificate that is currently throwing an OCSP error for me:

-----BEGIN CERTIFICATE-----
MIIGATCCBOmgAwIBAgISA0Dl/iAAWBYvXs32gJVo97waMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzEyMTUwNzMwNTlaFw0x
ODAzMTUwNzMwNTlaMBMxETAPBgNVBAMTCHZuZGIub3JnMIICIjANBgkqhkiG9w0B
AQEFAAOCAg8AMIICCgKCAgEAu+HSLW03hAr/pSDAuQWO534tIzN4Zd65sGmseH51
IVdwbAR+0sjqF0cS5DqkBOG7PZIvaySWjPTzrt7+IAHOLKRpRXpVuSvnBa+M7Iug
wmrFzTbmiwGh02jS5Vxr/izIb9bq++KhQ5P4sjxp13rMOVHUJTY6Mvi+XNnX9RGx
SbsFHAiHtO9vXmrtBx05ZtBetT6HLZiQ7hTf5CTi8VfTKJ8GTSiqvSmJBsMywwzH
vNvNp8YMrdms9rjcXmMIEpZc32uw5+IdswQpMQGCVvkYgEaKq/wt9C6byjMRVSSw
DUJTrOy3/XGWWtbCxW58b47+/YYcZ2yHAK93uM+PzngkxSEiGitj5mDbv2FHbN1A
4BSJK+3JadAniUsZuW1qPIEfr17MjluPxT4zUUq3Oasx2SqhxXUjsGSzc+0Yo/98
lMNPHluouLWVFEL2rjneOi0esN/s3bgBIKaPa8fgb1qX5hCueV4kayly9KP8IWMq
YMQ1EgBy+SFe5SMDmz65bPncah0hSY54pxDQPtB9sK2/Hr+Ms4mO8yYMBqhlHKYR
ICxMdVKRHdlfmSPMD3OsYuR0VBeWCzT9TeyP1SnR0MAbwwH3GWQ70HxvgV881duu
S6WYDRUApOscJqSThX1ESylrhR+hyMLoqYkfg/PF/nx70RhNesUsc180XRglVMOJ
KL0CAwEAAaOCAhYwggISMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEF
BQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUWONvZc10m6rk
WzIjXyIW8B4neA8wHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYI
KwYBBQUHAQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0
c2VuY3J5cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0
c2VuY3J5cHQub3JnLzAhBgNVHREEGjAYggh2bmRiLm9yZ4IMd3d3LnZuZGIub3Jn
MIH+BgNVHSAEgfYwgfMwCAYGZ4EMAQIBMIHmBgsrBgEEAYLfEwEBATCB1jAmBggr
BgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwgasGCCsGAQUFBwIC
MIGeDIGbVGhpcyBDZXJ0aWZpY2F0ZSBtYXkgb25seSBiZSByZWxpZWQgdXBvbiBi
eSBSZWx5aW5nIFBhcnRpZXMgYW5kIG9ubHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRo
ZSBDZXJ0aWZpY2F0ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlw
dC5vcmcvcmVwb3NpdG9yeS8wDQYJKoZIhvcNAQELBQADggEBAFZMDV47twMfTFfg
OyTD1LujmBNgJiP+K893d9bH1iS7Mm8YupFuI9KbTFPnZpcpj+2o7Qz5kP0GFpkK
baA/Iqd1TxsKNlqRSseiIg6aRNWS/4b5r+pDbtm7kJ1Zw/Op1OrT6fEA2MEJ2ke1
Q+R8Lzj+jsPmZWh9TkbzebbI62zcG4c+R1ZteXls8CMYVYqilb4EC2hbIRFMFEDv
LwcCoh21UuWhCg7EDbcrDADokJAIb7fnflfy4kJ2sbwPE/TONahdrCf8/k970I4l
DHgyJlvfG/2T71vQ5bcmK84NSEc8rmEL2naiKvOf/4UCFzAAHXcKrMCO/KK7oZYj
eel/U3A=
-----END CERTIFICATE-----

[stephenreay]

I'm seeing the same "unauthorized" error on two different domains (querying using openssl and https://certificate.revocationcheck.com).

I'm assuming this is the same issue reported at https://community.letsencrypt.org/t/ocsp-requests-returning-unauthorised/46820/6

Is there any update on a workaround, or a solution on LE's side for this?

[17dec]

I've been querying the OCSP today for the certificate that I posted, in order to measure the delay. I'm now finally getting a positive response, roughly 12 hours after getting the cert signed.

[stephenreay]

I wonder if that 12 hours is a combination of a delay, and an invalid cache rule - so it fails the first time due to being too soon, but the error response is cached for those 12 hours, despite the data being available upstream?

[jsha]

I'll take a closer look. 12 hours is indeed our caching period, and there's an open issue where unauthorized responses are cached for 12 hours. However, the root problem is that certificates should not be being returned to users until the OCSP response is ready. I'll see if I can reproduce

[jsha]

We've identified this as a replication lag problem and should have it resolved shortly: https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/5a3437ecdd1baf047386b2dd

[cpu]

@stephenreay @17dec @msanjay94 - The replication lag is resolved now and you shouldn't be experiencing the OCSP delays anymore. Can you folks confirm you're no longer seeing lagged OCSP responses for new issuance?

[17dec]

The errors were relatively sporadic in the first place, so I never had a reliable way to reproduce it. But to test it I've just renewed 19 certificates and immediately got a successful OCSP response on each of them, so things are looking good. Thanks.

Out of curiousity (I mean, I realize I'm using a free service and it's awesome that you're doing this at all!): Have steps been taken to prevent or quickly detect such replication lag in the future?

[cpu]

Out of curiousity (I mean, I realize I'm using a free service and it's awesome that you're doing this at all!): Have steps been taken to prevent or quickly detect such replication lag in the future?

Yup, the alert threshold has been adjusted. We were aware of the replication lag being large as a result of a separate ongoing maintenance but did not immediately realize that a user-facing service was affected - we moved that service while the maintenance is completed & will be more cognizant of this for future maintenance. Thanks!

[jsha]

And to add to what @cpu has said, we've updated our runbook collection to indicate what problems result from replication lag and how to address the lag quickly.

[stephenreay]

I've just requested the initial reg on 3 certs (I had stopped setup once the issue occurred, it wasn't urgent) and all returned good data during the subsequent OCSP request.

Thanks!

[msanjay94]

As @17dec said, I couldn't replicate the error myself in the first place, they were occurring in a random fashion. However, I've added retry+delay mechanism in my server, and haven't been noticing any issues since Saturday.

Thanks. :)