Boulder: TLS-SNI handshake error with Go 1.6 build

[phillipp]

Hi,
I have found an issue that correlates with the Go 1.6 build (problem started when it was deployed and was resolved after Rollback to Go 1.5):

We noticed that starting yesterday no TLS SNI challenge would succeed, despite trying a lot of times. The API error message was "Failed to connect to x.x.x.x:443 for TLS-SNI-01 challenge". There is a "status" code too that says 400. We first suspected firewall issued but figured out that the connection succeeds and the challenge certificate is presented to the client.

We confirmed that via the TLS terminator logs (we're using bud-tls): https://paste42.de/203a4bc665347f8782f95d4bd5909bdb/10906/

and openssl s_client:

https://paste42.de/ede21b0b7220483514eb70bf063b1e88/10905/

I noticed that the connection was closed by the TLS terminator after a "sslv3 alert bad certificate" / handshake failure.

Our currently configured protocols and cipers are listed here: https://www.ssllabs.com/ssltest/analyze.html?d=system.lima-city.de&s=91.216.248.35&latest2

We have not deployed any changes to the TLS terminator in the last months so I figured that a deploy on your side caused the problem and found out that it related to a deployment at the same day. After the rollback to Go 1.5 the problem disappeared.

Original post: https://community.letsencrypt.org/t/tls-sni-01-challenge-handshake-failure-after-11-08/18919

[cpu]

@phillipp Can you share which version of OpenSSL you are using on the server running bud-tls?

[cpu]

@phillipp Another q: Is system.lima-city.de the domain you were trying to issue for when you noticed the TLS SNI challenge failures? I don't believe it is - doesn't appear in ours logs recently.

[phillipp]

@cpu Sure, our bud-tls binary is compiled against openssl 1.0.2h.

We tried to issue certificates for ~50 different domain names, for example test.lechrock.de. system.lima-city.de was just an example if you need to check what cipher suites and/or protocols are enabled in out TLS setup but not one of the domains we tried to issue for. Both DNS names terminate on the same SSL setup, though.

[rolandshoemaker]

What software are you using to generate the certificates you are presenting to Boulder?

[phillipp]

We're using this ruby client to perform the challenge: https://github.com/unixcharles/acme-client

The certificate is then synced to the TLS terminators and the challenges verification requested. Starting yesterday, this always returned said error despite the exact same code working before and now again.

[rolandshoemaker]

@phillipp cool, thanks. Presumably you are not running that client on that same box as your TLS terminators, are you using the same version of OpenSSL on the client hosts?

The reason I ask is it appears that the problem here was your client is generating certificates that contain a encoding error that is indicative (as far as we can tell) of the combination of a specific range of OpenSSL versions and a client that doesn't explicitly set the version field on CSRs or certificates. Golang 1.6 added a much stricter parsing check for these encoding errors which caused the failures you observed.

While we are trying to figure out a way we can mitigate this problem on our end it would be extremely useful if you could, if possible, update your version of OpenSSL to the latest release, and keep up with updates to your client software. I'll go and check out unixcharles/acme-client and file a bug report for this problem.

Thanks for all your help!

[phillipp]

Yes, that is exactly right. These systems are using different distros and they don't provide the same SSL version, so that might be quite tricky. Is having the same openssl version the only way ore is there another workaround?

[rolandshoemaker]

There are two solutions, most likely on the client hosts you are using a OpenSSL version > 1.0.2, using any version above 1.0.2 should fix the issue, the other solution is to fix the problem in the client by explicitly setting the X509 version. unixcharles/acme-client already does this for the CSR but not for the self-signed certificates it generates for tls-sni-01 challenges. I've opened an issue for this fix, unixcharles/acme-client#97, once it is implemented and you update the problem should go away completely (whether or not you update OpenSSL).

[rolandshoemaker]

I'm also going to close this issue since this is not something we can really 'fix' in Boulder.

[phillipp]

Roland, thank you very much for your help!

I have submitted a PR for the ruby client. If we ever meet in real life remember me to buy you a beer ;-)

[jsha]

Yep, props to @rolandshoemaker for figuring this out quickly. Also thanks @phillipp for the very thorough and helpful bug report.

[phillipp]

@jsha

The problem popped up again, today. The challenge says:

"invalid", {"type"=>"urn:acme:error:connection", "detail"=>"Failed to connect to 91.216.248.25:443 for TLS-SNI-01 challenge", "status"=>400}

The request to the SSL server is made and then the clients resets the connection after receiving the certificate:

Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend new
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend after read_cb() => 219
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend ssl_cert_cb {0}
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SSL_read() => -1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend connect 0
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend ssl_cert_cb {1}
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SSL_read() => -1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SNI name found: "ff63335bbefe493709edc2018b3ad0a1.d65d051d1780bdbe7f319b859dd34b51.acme.invalid"
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend ssl_cert_cb {2}
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SSL_read() => -1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend uv_write(1159) iovcnt: 1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend immediate write
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend write_cb => 1159
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend read_start
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend recycle
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SSL_read() => -1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend after read_cb() => -104
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend SSL_read() => -1
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend closed because: uv_read_start(client) cb returned -104 (connection reset by peer)
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend force closing (and waiting for other)
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on backend force closing (and waiting for other)
Nov 23 15:48:16 zeus bud[25090]: client 0x2185e80 on frontend close_cb

Is there another breaking change in the challenge process?

[phillipp]

I have just seen that our system requested (successfully) two other certificates, the problem seems only to occur with that one domain name. Is it possible that there is some problem/edge case with the private key or the CSR?

[cpu]

Hi @phillipp,

I believe you're running into the same bug as before. We've switched to using Go 1.7 for our builds following the announcement we were going to do so in November. At this point we can no longer support pre-1.0.2 OpenSSL clients.

Is it possible that the two certificates that were issued successfully had valid un-expired authorizations already associated with your account but the failing domain did not and thus had to perform a TLS-SNI-01 challenge that didn't succeed due to the OpenSSL client bug?

[phillipp]

@cpu I have checked (and to be on the side auf caution, upgraded die acme lib), and the version of both CSR and certificate is 2. Still not working.

[phillipp]

But you are right, the successfully generated certificates seem to have skipped auth because they already had been authorized before.

But judging from the logs it appeared directly after this maintenance, which I assume was the "hard cutoff":

https://letsencrypt.status.io/pages/maintenance/55957a99e800baa4470002da/582d6b53fbdf3f1a1400002c

Am I missing anything? There is two options:

Either a) we don't generate the right CSR/self signed cert, or b) there is more that needs to be done than just setting the certificate version.
I think it is not a because I've verified that the CSR and self signed certificate both have a version that is "2". Is that not right?

[cpu]

@phillipp That maintenance matches up to when I believe operations made the switch from Go 1.5 -> Go 1.7 for the hard cut off. The authorization reuse also explains why some of your domains passed, that's :+1:, one less mystery.

Am I correct in assuming that you are still using two systems, with differing OpenSSL versions, in the process of end-to-end creating a certificate? What are the two versions?

Either a) we don't generate the right CSR/self signed cert, or b) there is more that needs to be done than just setting the certificate version.
I think it is not a because I've verified that the CSR and self signed certificate both have a version that is "2". Is that not right?

Its quite puzzling. I'm only aware of A as a potential cause but its always possible we're overlooking another compatibility issue. We're short staffed right now for the American Thanksgiving holiday and the individuals with the most context are out for the next few days. I'll try and take a closer look from our end but I suspect we might need to rope in more eyes to get to the bottom of it.

[phillipp]

Yes, we're still using different systems. The server that is making the authorization is using libssl-dev_1.0.1-4ubuntu5.38 and the servers that do SSL termination use 1.0.2j.

[jsha]

The underlying error we're seeing in our logs is still: tls: failed to parse certificate from server: asn1: structure error: empty integer.

Here's an example Go program that parses a PEM formatted certificate: https://play.golang.org/p/IglCG8JSLG. Can you paste your generated certificate in place of the example one, and see if it parses correctly? If not you can make the necessary tweaks to acme-charles and/or upgrade OpenSSL, then check that the result parses correctly with the Golang example.

[phillipp]

That helped me a lot to iterate faster on the problem, thanks!

I have found out that it's the serial number of the certificate. I've added the serial number "1" and it works again. Pull request has been opened for the library: https://github.com/unixcharles/acme-client/pull/106

Thanks again for all the help, keep up the good work!

[cpu]

@phillipp Thank you for doing the leg work to find the root cause. I really appreciate it! I've added a note to the community forum post for the larger OpenSSL issue so that hopefully other client developers are able to fix this issue as well.

Take care!