Boulder: False "Incorrect validation certificate for tls-alpn-01 challenge"

[tlhackque]

Although not required by the protocol, most tls-alpn-01 validation certificates are signed by the issuer (ACME client).

If the key is too large, LetsEncrypt reports Missing acmeValidationV1 extension.

This is not true - and required a lot of debugging. While there's no need to accept huge certificates, the error message should be more like "Validation certificate too large: try signing with a smaller key." (There is no advantage to using a stronger key, as the payload itself is what provides security.)

I saw this failure with a 4096-bit RSA key (because the client used the same key type/size that was requested in the CSR.)

Examples.:
Failing certificate (note that the proper extension - 1.3.6.1.5.5.7.1.31: is present:
````
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
d4:50:23:09:8e:30:c0:03:dd:c6:69:ab:2c:15:f3:2b:e0:18:c7:bb
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = wikiworld.litts.net
Validity
Not Before: Mar 19 20:08:59 2020 GMT
Not After : Mar 26 20:08:59 2020 GMT
Subject: CN = wikiworld.litts.net
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:b5:f7:35:ea:cd:be:4f:f6:e4:2c:9a:d2:8b:54:
75:a6:bc:d4:cc:22:9a:13:38:57:7c:52:d4:67:41:

                d8:ea:cd:85:84:6d:d5:9c:4b:1e:0c:f8:bf:5c:bc:
                8f:72:0f:9b:82:67:5a:73:47:c2:3b:e8:cd:42:34:
                7b:a7:48:82:be:b8:bc:bd:51:33:09:e4:5d:e4:35:
                aa:f5:9d
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Basic Constraints: critical
            CA:FALSE
        X509v3 Subject Alternative Name:
            DNS:wikiworld.litts.net
        1.3.6.1.5.5.7.1.31: critical       <-------------
            . .qD..I............5.K......i..|l
Signature Algorithm: sha256WithRSAEncryption
     7f:01:bf:7e:b8:8c:00:8f:bf:2e:3c:28:f2:4a:a5:4f:8e:54:
     e4:67:67:f5:12:f1:db:05:76:e3:d0:00:75:e4:5b:64:a1:3e:
     1e:4b:c3:6c:fd:83:32:50:78:7a:a0:7f:84:67:1a:e5:d9:52:

````
The certificate was captured just before going into the SSL protocol engine.

Logged response from LE (one of many - may not be from the above cert):
{\n "identifier": {\n "type": "dns",\n "value": "wikiworld.litts.net"\n },\n "status": "invalid",\n "expires": "2020-03-26T20:08:54Z",\n "challenges": [\n {\n "type": "tls-alpn-01",\n "status": "invalid",\n "error": {\n "type": "urn:ietf:params:acme:error:unauthorized",\n "detail": "Incorrect validation certificate for tls-alpn-01 challenge. Missing acmeValidationV1 extension.",\n <------------------ "status": 403\n },\n "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/44567922/UhYA6w",\n "token": "yC55HQ8n0mNFy1pdBKXBE-qOhqGrGBzVUQW-MOz4r5M",\n "validationRecord": [\n {\n "hostname": "wikiworld.litts.net",\n "port": "443",\n "addressesResolved": [\n "96.233.62.58",\n "2001:470:8f95:941::81"\n ],\n "addressUsed": "2001:470:8f95:941::81"\n }\n ]\n }\n ]\n }

[alexzorin]

If the key is too large, LetsEncrypt reports Missing acmeValidationV1 extension.

How did you come to that conclusion? I just tried the TLS-ALPN challenge using an 8192-bit RSA key, and it worked fine.

Can you post a private key and TLS-ALPN certificate that you know will fail?

[tlhackque]

How did you come to that conclusion?

Debugging unrelated changes in mod_md (the apache ACME client), I got the error responses from LE's staging CA, posted above.

From the message - which I posted above - I thought mod_md must be supplying the wrong certificate. I knew that the validation cert had the extension. (It's visible on disk; I'd also recently been in the code.) I also "knew" that nothing I did could possibly break this. But I had to assume I was guilty until proven otherwise.

So, I instrumented the software, and had the bits passed to OpenSSL written to a PEM file at the point they're passed to the library (in mod_ssl, if you want to be precise). That would be what I pasted above.
and attached here.

I did not capture the private key. The actual transport path between creation and delivery is a maze of twistry passages - but they move as a pair. I suppose that the ASN/1 encoding of the serial number, validation extension, or public key could move the extenion around a bit, depending on how it compresses. The validator never sees the private key - one hopes.

Without making any other change, I told mod_md to generate a 2048 bit key. The errors went away.
This is the captured certificate.

I wanted to belive it was my code (or the mod_md owner's :-). I went as close the the wire as I could to prove it & bisect. I should also note that mod_mds default is 2048 bit RSA - and I was using tls-alpn-01 for the first time. As it happened, on a machine that uses the larger keys. As it turns out, the evidence points to a the LE server.

Finally, this didn't happen just once. I have logs full of traces showing it happening for at least several hours - multiple attempts, and starting over generating new challenges / validation certs.

I also looked at the boulder VA code. It seems pretty simple - it gets a certificate and looks for the extension. Falls off the end and says "no extension'. Didn't seem like it could go wrong - but there I was.

The only conclusion I could come to was that the underlying code truncates the validation certificate - perhaps to avoid denial of service attacks. After all, the validator does not need the signature - it's there because clients generate one for any other certificate, and they don't know that this one is special. So functionally, chopping off the signature and ignoring it would have no effect.
In any case, if the VA's library truncated the receive certificate, increasing the public key size could push the magic extension past the point of truncation. The validation extension happens to be the last thing before the signature,,,

That part is just a theory - I'm not a GO or a Boulder person. But what I observed is pretty clear - 4096-bit key: fails. 2048-bit key does not. Perhaps there's a better explanation. I'm all ears :-)

In the meantime, I'm patching mod_md to decouple the key type/strength of the validation certificates from that of the CSR/issued certificates...there is no point to creating/transporting a large/strong signature that adds no value.

[alexzorin]

In an effort to reproduce your problem, I also tried building Apache 2.4.41 + mod_md from master just now. Here is the configuration I used:

SSLProtocol TLSv1.2
MDCertificateAuthority https://acme-staging-v02.api.letsencrypt.org/directory
MDCertificateAgreement accepted
MDContactEmail <redacted>

<MDomain brazil-rachel.bnr.la>
  MDCAChallenges tls-alpn-01
  MDPrivateKeys RSA 4096
  MDStapling on
</MDomain>

<VirtualHost *:443>
  ServerName brazil-rachel.bnr.la
  Protocols h2 http/1.1 acme-tls/1
  SSLEngine on
  DocumentRoot /usr/local/apache2/htdocs
</VirtualHost>

Before starting Apache, I started up a Wireshark to capture all the 443/tcp chatter.

Starting Apache, the certificate issued successfully.

I dumped the certificate from the packet capture and the certificate looks just like the one you uploaded:

 Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            bb:de:10:81:56:65:d0:8a:39:d7:0f:94:41:b9:84:93:1c:d8:6e:c0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = brazil-rachel.bnr.la
        Validity
            Not Before: Mar 20 04:02:40 2020 GMT
            Not After : Mar 27 04:02:40 2020 GMT
        Subject: CN = brazil-rachel.bnr.la
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:ea:7c:ce:9e:f6:fe:5d:fa:ba:75:48:91:90:1c:
                    52:03:1e:e4:84:bd:78:04:9e:21:65:35:d3:b1:e9:
                    5c:46:d5:8f:b1:6c:37:cb:7f:ee:e3:86:01:2d:8e:
                    79:b3:51:67:2e:63:01:4a:0c:2d:12:d3:a7:d0:00:
                    f3:c8:76:32:38:b9:62:24:76:97:6b:22:28:4d:a3:
                    c2:77:97:07:c0:b0:4d:56:7c:45:96:9e:42:2e:1d:
                    80:8a:4e:6f:50:25:bc:e6:c4:ff:cb:30:dc:e7:ef:
                    b6:e9:78:7a:75:53:49:d3:9e:d6:25:65:30:6c:b7:
                    40:3c:08:41:6b:15:c1:a5:75:3f:ab:e5:ed:ad:f5:
                    36:32:ee:fd:d9:6a:f6:ff:89:3c:a1:5c:b0:87:32:
                    a6:46:2f:35:9c:1c:fd:a5:ee:ae:8c:e6:d8:64:ed:
                    28:09:6e:01:f6:f2:43:9a:ff:38:17:f4:64:8b:c2:
                    19:d4:6e:03:b8:e7:65:b6:06:f1:b0:4e:01:60:0d:
                    73:43:37:b5:96:d2:fd:27:e7:f5:8a:ed:7b:d1:8a:
                    79:10:df:b4:f1:65:94:5c:49:30:e7:e7:86:53:3e:
                    af:d1:74:97:ad:53:4d:dd:47:31:72:82:67:c1:ee:
                    a7:c7:81:52:5c:cc:8f:1f:9d:73:4b:ee:d9:92:98:
                    c3:bd:8e:34:e8:45:08:ea:f8:8b:c6:ca:54:6f:f0:
                    b5:dd:99:d2:b9:33:43:4a:5e:ec:30:37:cd:b7:7a:
                    df:bc:1e:00:c1:b1:79:da:d0:fc:42:84:93:b8:b7:
                    c1:fa:e1:d0:bf:e9:8f:92:08:ac:bd:60:05:b8:8f:
                    7c:ad:1f:8c:2b:8a:91:fc:bf:65:94:87:80:21:e1:
                    50:6c:ac:c4:8b:b3:be:ea:a3:a5:1a:95:b8:c7:2b:
                    4c:40:ab:64:85:c9:a7:b5:4e:05:f4:c5:22:63:66:
                    31:ed:75:32:80:a4:77:b4:c0:7b:7d:ac:c1:70:27:
                    85:8b:ad:2a:f6:37:d5:22:83:ad:6f:38:30:c2:37:
                    e7:9c:28:c3:ea:ac:00:ff:04:97:c1:86:e1:35:0c:
                    4e:74:1a:3f:85:e9:8c:29:8d:fd:77:d3:26:80:c3:
                    f7:93:c4:b2:54:2f:f5:89:da:9a:c6:c5:03:21:bf:
                    41:68:10:88:1f:6b:46:25:b3:b9:9c:7a:97:bf:e4:
                    e6:ba:69:1e:3b:4b:03:c4:64:15:a1:82:b8:c8:75:
                    83:57:e8:0f:fd:c7:b6:bb:cc:46:67:c6:2a:c2:64:
                    ff:21:fe:32:e3:f0:8e:30:14:d6:5d:05:af:b1:41:
                    9f:4c:f7:52:ff:07:5e:df:87:32:28:22:13:ba:c3:
                    cc:56:3d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:brazil-rachel.bnr.la
            1.3.6.1.5.5.7.1.31: critical
                . .;B.Zek.$mW~<...
........]......
    Signature Algorithm: sha256WithRSAEncryption
        1c:6b:1f:6e:15:16:80:04:67:31:91:56:9c:40:5b:cb:da:6e:
        fc:3c:ee:90:b7:79:a3:73:ba:93:9f:e2:f6:c4:d3:52:1e:47:
        e7:de:96:be:5c:45:5e:bd:7a:dc:27:ed:23:94:a3:0c:7f:e4:
        7e:5c:71:5f:f3:97:88:d2:a5:80:78:e3:d8:4d:45:db:b1:bf:
        fe:34:ff:da:16:30:94:93:99:57:ad:15:df:07:6a:7a:7d:a0:
        67:27:be:8c:51:20:b1:a1:2e:51:11:cc:4f:fa:dc:f5:81:f4:
        6f:df:0c:13:da:c0:df:bb:db:a7:fe:ec:32:7f:66:fc:e2:b4:
        65:a1:cb:8a:55:78:5a:bd:20:89:0b:90:de:24:8f:73:88:dc:
        2a:88:6d:c3:08:6f:01:5c:69:46:3d:43:b9:cf:92:b1:f9:11:
        05:99:c3:52:71:b2:12:ab:df:1d:80:c7:dd:ba:29:88:d1:4c:
        02:b1:6c:04:1c:28:65:1c:4c:91:d1:15:af:ee:11:09:b9:78:
        27:51:20:bb:54:ea:58:2a:71:be:f0:63:e8:79:42:b5:cb:24:
        f9:69:9c:45:89:b0:49:08:77:58:37:bc:e6:b1:8c:02:73:ac:
        bb:18:19:1e:eb:85:ff:87:16:bc:6d:2e:ab:59:50:84:1e:75:
        59:c6:0b:06:4b:c6:33:dd:c9:d1:d2:e8:60:47:f6:ee:c1:03:
        31:2c:42:4a:55:bd:49:90:59:6f:8c:0e:f1:89:6f:bb:c4:bd:
        2e:25:e5:09:d4:08:75:1d:08:86:9d:3e:3f:7e:c3:36:f8:db:
        df:84:56:5c:9f:f5:64:de:ef:69:c5:85:91:66:ed:34:1e:95:
        15:86:f2:c7:b6:c3:05:0d:aa:eb:e7:e7:ad:22:c2:d1:56:36:
        e2:20:0d:38:bf:bf:93:28:3a:47:75:da:01:f8:35:00:d8:2b:
        ec:0c:a2:2b:ee:b9:2e:ed:5f:a1:a8:29:77:25:a5:1b:4d:0d:
        00:59:1f:24:05:b7:e1:d1:f7:79:30:03:7d:a2:65:c0:16:c3:
        a7:3d:50:60:32:51:39:45:0c:30:23:0b:77:83:4d:f5:3f:78:
        a6:0d:51:da:17:ee:98:20:ee:7e:05:38:bb:97:95:a8:a6:a2:
        c6:23:61:b0:20:ae:26:4d:a8:d9:ec:51:67:70:7f:e9:dd:7e:
        ae:e2:70:4f:39:59:1f:08:e5:09:2f:15:59:b3:0e:b8:11:e1:
        91:04:82:6d:bc:e7:2a:ee:35:a4:6a:db:d0:4d:68:07:5f:08:
        ac:7b:38:15:35:01:19:8d:c5:93:a6:fc:bc:aa:e8:8b:a4:0a:
        e9:14:a9:fd:e0:ea:aa:7e

Its size is very very close to the certificate that you posted.

Any chance you can do a full-take packet capture of port 443, with a blank mod_md state and Apache stopped, and then starting Apache? (You will also have to first force TLS 1.3 to be disabled, as important parts of the handshake became encrypted in that version).

I really suspect that there is a bug in mod_md that causes it to not actually respond with the validation certificate (under some unknown circumtances). Or something else weird is happening in your environment.

A packet capture would go a huge way towards aiding understanding and a reproduction (if any) of your problem.

[rolandshoemaker]

I took a look at our logs, the certificate being presented to our validation server was not the one you posted, but a certificate with a Apache Managed Domain Fallback common name and no acmeValidationV1 extension. It looks like there are two addresses for your domain, an IPv4 address and an IPv6 address, and we are using the IPv6 address for validation. It may be the case that you're only presenting the TLS-ALPN certificate on the IPv4 address and Apache is presenting some other default certificate on the IPv6 address.

[tlhackque]

I took a look at our logs, the certificate being presented to our validation server was not the one you posted, but a certificate with a Apache Managed Domain Fallback common name and no acmeValidationV1 extension. It looks like there are two addresses for your domain, an IPv4 address and an IPv6 address, and we are using the IPv6 address for validation. It may be the case that you're only presenting the TLS-ALPN certificate on the IPv4 address and Apache is presenting some other default certificate on the IPv6 address.

Thank you for the information from your logs. It's very helpful. I'll go back to digging on my end.

Indeed, the "Fallback certificate" is generated by mod_md. We use it (at least in theory) only to send "service unavailable" when we don't have a real one from you. My first thought was that we might be sending it, which is what sent me down the maze. But we definitely weren't - up to the point that I could see (including with wireshark - the cert is encrypted.) I didn't think of forcing TLS 1.2.

The capture came from the query to the IPv6 address, so that's not root cause. Both addresses go to the same httpd instance, and are treated identically.

I also see the protocol switch to acme-tls/1, which is what causes the response.

If a connection were cached (which it shouldn't be), no cert would be sent. We are, of course, polling for completion.

Disabling TLS1.3 sounds like a good idea. I'll see where that takes me.

Do your logs show all the validation attempts/responses from this machine? If it's not too much trouble, it might help to see where in the sequence mod_md failed to respond if I could match them up with mine...

Thanks for the help. Although I can't explain why changing the keysize made a difference in our behavior, I'm actually glad to see the evidence pointing back at mod_md. It's easier to work on.

I'll let you know when I've gotten to the bottom of this.

[tlhackque]

There appears to be a race condition. It reproduced first try with TLS1.2 - thanks for that. I have a trace, which shows mod_md responding with APLN Next Protocol: acme-tls/1 - and in the same record, the fallback certificate! The issue lies somewhere between mod_md, mod_ssl, and OpenSSL...

I will get to the bottom of this - clearly in mod_md's court. Sorry for the false alarm - but your help is greatly appreciated!

[tlhackque]

The problem is well understood, and is contained within the webserver.

The issue is that mod_md was recently updated to support dual certificates (e.g. simulataneous RSA & EC). When two certificates are configured and no certificate from the CA is available, it creates and preloads two "fallback certificates".

These are replaced when the CA certificates issue.

When a tls-alpn-01 challenge comes in, a validation response certificate is created and loaded.

Unfortunately, it displaces only one of the fallback certificates. Depending on how the TLS negotiation turns out, the CA receives either the validation response, or the remaining fallback certificate. That depends on configuration of cipher preferences on both sides.

Due to the way interfaces are defined between mod_md, mod_ssl, and OpenSSL, while the fix is conceptually simple, it's tricky to implement so that all combinations of versions will do something reaasonable.

However, I'm confident that this is the real answer.

Thank you both for your help. Knowing what was in your logs was an important clue. And the reminder that I could actually see things by forcing TLS1.2 was also key.

[rolandshoemaker]

Makes sense, thanks for debugging this so thoroughly on your end!