Boulder: IPv6First with a hanging ipv6 connection never tries the ipv4 address

[marktheunissen]

Steps to recreate:

• Attempt to go through http-01 validation for testboulderv6.ddns.net
• Validation fails, even though there is a valid A record.

Details:

Given a domain with both A and AAAA records, if the connection to the AAAA record hangs, we expect that the A record is tried after a 5 second timeout given by singleDialTimeout.

Instead, the A record is never tried.

$ dig +short AAAA testboulderv6.ddns.net
2001:4801:7901:0:aff8:a2d8:0:249
$ dig +short A testboulderv6.ddns.net
23.185.0.3

$ curl -4 testboulderv6.ddns.net/.well-known/acme-challenge/abcdef
abcdef.pubkey

```
$ time curl -6 -v testboulderv6.ddns.net/.well-known/acme-challenge/abcdef

• Trying 2001:4801:7901:0:aff8:a2d8:0:249...
• connect to 2001:4801:7901:0:aff8:a2d8:0:249 port 80 failed: Connection timed out
• Failed to connect to testboulderv6.ddns.net port 80: Connection timed out
• Closing connection 0
  curl: (7) Failed to connect to testboulderv6.ddns.net port 80: Connection timed out

real 2m7.196s
user 0m0.002s
sys 0m0.008s

Here's what I see in the challenge:

```json
{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:connection",
    "detail": "Fetching http://testboulderv6.ddns.net/.well-known/acme-challenge/<redacted>: Timeout",
    "status": 400
  },
  "uri": "<redacted>",
  "token": "<redacted>",
  "keyAuthorization": "<redacted>",
  "validationRecord": [
    {
      "url": "http://testboulderv6.ddns.net/.well-known/acme-challenge/<redacted>",
      "hostname": "testboulderv6.ddns.net",
      "port": "80",
      "addressesResolved": [
        "23.185.0.3",
        "2001:4801:7901:0:aff8:a2d8:0:249"
      ],
      "addressUsed": "2001:4801:7901:0:aff8:a2d8:0:249",
      "addressesTried": []
    }
  ]
}

http01Dialer.Dial() only checks that it can CONNECT to the target socket on that IP, not that the server is actually alive, is an HTTP server, and/or is capable of responding to any request. https://github.com/letsencrypt/boulder/blob/master/va/va.go#L231-L236

Ideally, it would be nice if the test was stricter so that we failover to the ipv4 if the v6 address is broken. How about performing a round-trip, and ensuring a status code or some valid HTTP headers are received?

Ipv6 is poorly supported (e.g. not available in Google Cloud), so it's quite a pain to pre-verify these domain records before requesting a cert is issued.

If this is not possible, a feature request might be to add a parameter so that the client can request whether ipv6 or v4 should be tried first.

[cpu]

Hi @marktheunissen,

Thanks for the very detailed bug report & reproduction. I really appreciate it. This particular feature has been giving me the gears :sweat:

If this is not possible, a feature request might be to add a parameter so that the client can request whether ipv6 or v4 should be tried first.

In general I'm hesitant to add knobs to the verification process that allows the requester to change how we validate a domain. It's likely to give more leverage to an attacker and increases complexity with more moving pieces.

I think your suggestion to make the initial IPv6 test fail for a broken server instead of just an unreachable server are the best path forward for this issue.

[marktheunissen]

Hi @cpu, thanks for getting back so quickly, yeah doing an HTTP req/resp cycle, regardless of the code or headers returned, might be the easiest.

[jsha]

Hi @marktheunissen! We talked about this in a little more detail and decided against implementing it. There are a couple of possible cases where a server could fail on IPv6 and not IPv4:

1. IPv6 routing is randomly broken -> timeout from some perspectives, success from others.
2. A server listens differently on IPv6 than IPv4, and the IPv6 listener is broken in some way (e.g. connects, but doesn't reply to requests, or replies differently than the IPv4 listener).

I think we want to be resilient to (1), because it's a common problem with IPv6 in general, and is often out of the control of the server operator trying to get a certificate (although they should consider removing their AAAA record). However (2) is less common IMO, and is within the control of the server operator, so I think we should probably not add the complexity necessary to work around it. So I'm considering this a "won't fix" for now. But I really do appreciate your taking the time to file such a detailed bug report!

[marktheunissen]

Thanks @jsha, I understand the reasoning.

From our perspective, we act as a broker between our customers and Let's Encrypt. We don't control the domain DNS, but attempt to issue certificates for those domains. So we want to be reasonably sure that our http-01 verification requests succeed before we ask Let's Encrypt to make them, as there can be thousands of domains pending and we don't want to hammer the service and get rate limited or just be a nuisance.

We've seen a lot of (2) - customers will set their ipv6 records to any random thing, but never notice that it doesn't work, because http clients like browsers still fall over to ipv4.

We try to pre-verify all domains on the cert, before getting Let's Encrypt to verify, but this means we're duplicating Boulder's logic and this is a difficult task. With ipv6, it's especially difficult to pre-verify the domain is correct and serving challenge responses from the platform, because most cloud providers don't actually support outbound ipv6 connections. We've got a workaround now, but it involves an ipv6 proxy.

Other issues we encounter are domains that are blacklisted by LE, or appear on the Google malware blacklist, or their DNS has invalid CAA records, so there are quite a few things to consider. :)

Thanks for looking into this one.

[marktheunissen]

We could use staging to pre-verify but we've seen some warnings about staging being unstable.

[teohhanhui]

Why does it matter when the request over IPv6 timed out, whether it's because of an intermittent connection / routing issue, or due to a misconfiguration? (How do you tell the difference?)

A timeout is a timeout, and it should fall back to IPv4, no?