Caddy: Cloudflare DNS challenge times out after 2 minutes

[sfkpmr]

1. What version of Caddy are you using (caddy -version)?

0.11.1

2. What are you trying to do?

Use DNS challenges with Cloudflare.

3. What is your entire Caddyfile?

Example:

www.domain.com domain.com {
        gzip
        tls {
        dns cloudflare
        }
        proxy /site1 IP:port {
                transparent
                }
}

site2.domain.com {
        tls {
        dns cloudflare
        }
        proxy / IP:port {
        transparent
        }
}

Caddy works fine when removing TLS/DNS tags and using HTTP challenges.

4. How did you run Caddy (give the full command and describe the execution environment)?

systemctl start caddy

(/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp)

CentOS, Caddy installed according to this.

5. Please paste any relevant HTTP request(s) here.

-

6. What did you expect to see?

DNS challenge passing.

7. What did you see instead (give full error messages and/or log)?

Activating privacy features… [INFO][sub.domain.com] acme: Obtaining bundled SAN certificate
[INFO][sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/…
[INFO][sub.domain.com] acme: Could not find solver for: tls-alpn-01
[INFO][sub.domain.com] acme: Could not find solver for: http-01
[INFO][sub.domain.com] acme: Trying to solve DNS-01
[INFO][sub.domain.com] Checking DNS record propagation using [DNS]

Caddy waits for two minutes, then restarts and does the same thing again.

The TXT is written to Cloudflare and remains there until it's removed manually.

8. How can someone who is starting from scratch reproduce the bug as minimally as possible?

Similar issue described here.

Lego has environment variables that seem to handle this problem.

Caddy forum discussion.

[wwarodom]

I got similar bugs too. HTTPS works fine without Cloudflare.

[francislavoie]

Could either of you try from the latest on master? There may be some other issues, but it would be good to see if the cloudflare provider works. Since 0.11.1, there's been some fixes to that provider, such as https://github.com/xenolf/lego/pull/721. 0.11.2 should have those fixes as well, but has some other unresolved issues, so I recommend trying master.

Edit: ack, fat fingered the close and comment button. Sorry

[sfkpmr]

The issue remains, but the log looks different in 0.11.2, and directly lists the 2 minute timeout:

[INFO] [sub.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/XXX
[INFO] [sub.domain.com] acme: Could not find solver for: tls-alpn-01
[INFO] [sub.domain.com] acme: Could not find solver for: http-01
[INFO] [sub.domain.com] acme: use dns-01 solver
[INFO] [sub.domain.com] acme: Preparing to solve DNS-01
[INFO] cloudflare: new record for sub.domain.com, ID [X]
[INFO] [sub.domain.com] acme: Trying to solve DNS-01
[INFO] [sub.domain.com] acme: Checking DNS record propagation using [DNS]
[INFO] Wait [timeout: 2m0s, interval: 2s]
[INFO] [sub.domain.com] acme: Waiting for DNS record propagation.

Last line repeating every 2 seconds until Caddy restarts after hitting the 2 minute timeout.

Updated with curl https://getcaddy.com | bash -s personal tls.dns.cloudflare.

[sfkpmr]

Tried 0.11.3 too, no change.

For completeness, before Caddy restarts it says "[INFO] [sub.domain.com] acme: Cleaning DNS-01 challenge".

[francislavoie]

@mholt any ideas?

[whitestrake]

Just built 0.11.3 myself and can't replicate the issue.

whitestrake in ~ at kairos is 🐳 v18.09.1
➜ dc exec caddy caddy -version
Caddy 0.11.3 (+2ea544e Tue Feb 05 23:27:44 UTC 2019) (unofficial)
1 file changed, 2 insertions(+)
caddy/caddymain/run.go

(the modifications are for http.git and tls.dns.cloudflare)

caddy_1  | Activating privacy features... 2019/02/05 23:33:38 [INFO][FileStorage:/root/.caddy] Started certificate maintenance routine
caddy_1  | 2019/02/05 23:33:39 [INFO] acme: Registering account for [email protected]
caddy_1  | 2019/02/05 23:33:39 [INFO] [kairos.whitestrake.net] acme: Obtaining bundled SAN certificate
caddy_1  | 2019/02/05 23:33:40 [INFO] [kairos.whitestrake.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/eG0enmAnm8wl0JblKbRa7z897g_tA8BXOgIs4bVCrA8
caddy_1  | 2019/02/05 23:33:40 [INFO] [kairos.whitestrake.net] acme: Could not find solver for: tls-alpn-01
caddy_1  | 2019/02/05 23:33:40 [INFO] [kairos.whitestrake.net] acme: Could not find solver for: http-01
caddy_1  | 2019/02/05 23:33:40 [INFO] [kairos.whitestrake.net] acme: use dns-01 solver
caddy_1  | 2019/02/05 23:33:40 [INFO] [kairos.whitestrake.net] acme: Preparing to solve DNS-01
caddy_1  | 2019/02/05 23:33:41 [INFO] cloudflare: new record for kairos.whitestrake.net, ID 255d406db72367697fba415cba7d8daf
caddy_1  | 2019/02/05 23:33:41 [INFO] [kairos.whitestrake.net] acme: Trying to solve DNS-01
caddy_1  | 2019/02/05 23:33:41 [INFO] [kairos.whitestrake.net] acme: Checking DNS record propagation using [127.0.0.11:53]
caddy_1  | 2019/02/05 23:33:41 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
caddy_1  | 2019/02/05 23:33:41 [INFO] [kairos.whitestrake.net] acme: Waiting for DNS record propagation.
caddy_1  | 2019/02/05 23:33:43 [INFO] [kairos.whitestrake.net] acme: Waiting for DNS record propagation.
caddy_1  | 2019/02/05 23:33:46 [INFO] [kairos.whitestrake.net] The server validated our request
caddy_1  | 2019/02/05 23:33:46 [INFO] [kairos.whitestrake.net] acme: Cleaning DNS-01 challenge
caddy_1  | 2019/02/05 23:33:47 [INFO] [kairos.whitestrake.net] acme: Validations succeeded; requesting certificates
caddy_1  | 2019/02/05 23:33:48 [INFO] [kairos.whitestrake.net] Server responded with a certificate.
caddy_1  | done.
caddy_1  | https://kairos.whitestrake.net
caddy_1  | 2019/02/05 23:33:48 https://kairos.whitestrake.net
caddy_1  | http://kairos.whitestrake.net
caddy_1  | 2019/02/05 23:33:48 http://kairos.whitestrake.net

@sfkpmr:

I have seen in the past people encountering this issue with some domains but not others. If you have another domain to test with, I'd suggest that (you can get free short-renewal .tk domains).

You could also open your Cloudflare dashboard and look at the domain's DNS records while this challenge is happening to double check that the record is being added. If you can see it in your dashboard, you can get your Cloudflare nameserver and ensure it's responding properly by running dig @[NAMESERVER] _acme-challenge.subdomain.example.com txt (replace [NAMESERVER] as appropriate).

[magikstm]

@sfkpmr What is the domain name that has the issue?

[sfkpmr]

@whitestrake @magikstm

It's a .co domain, could that be the issue?

[whitestrake]

I highly doubt the TLD itself is the issue.

The thing I mentioned was just Cloudflare being on the fritz for some specific domains on its service, i.e. working for bar.example.com but not for foo.example.com.

I assume magikstm was asking after the actual domain name, so that we can investigate it for any discrepancies that might contribute to the issue.

[magikstm]

Could it be a misconfiguration because it uses the Cloudflare certificate and not the Let's Encrypt one?

https://crt.sh/?q=wallen.co&dir=^&sort=1&group=none
https://www.geocerts.com/ssl-checker

Could you revoke the Cloudflare one, use it explicitly or use the Let's Encrypt one explicitly?

[magikstm]

I think you may need to adjust your TLS to force it to use your Cloudflare certificate:
tls ../cert.pem ../key.pem
Or

tls {
    load /www/certificates
}

https://caddyserver.com/docs/tls

[sfkpmr]

Could you revoke the Cloudflare one, use it explicitly or use the Let's Encrypt one explicitly?

I've disabled Cloudflare's SSL and will try again tomorrow. I assume Geocerts should flip to Let's Encrypt/nothing.

[riker09]

Maybe I can add something to this ticket. I have to domains example-a.com and example-b.org. example-b.org has a wildcard (*) CNAME record that points to example-a.com as well as CNAME example-b.org pointing to example-a.com. Here's an excerpt from the zone file

[...]
;; CNAME Records
*.example-b-org.    120 IN  CNAME   example-a.com.
example-b-org.  120 IN  CNAME   example-a.com.

The zone file for domain example-a.com is this:

;; A Records
*.example-a.com.    120 IN  A   1.2.3.4
example-a.com.  120 IN  A   1.2.3.4

I can issue certificates for *.example-a.com but __not__ for *.example-b.org. Here's part of the logfile:

2019/02/28 15:15:53 [INFO] [foo.example-b.org] acme: Trying to solve DNS-01
2019/02/28 15:15:53 [INFO] [foo.example-b.org] acme: Checking DNS record propagation using [127.0.0.11:53]
2019/02/28 15:15:53 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2019/02/28 15:15:53 [INFO] [foo.example-b.org] acme: Waiting for DNS record propagation.
2019/02/28 15:15:55 [INFO] [foo.example-b.org] acme: Waiting for DNS record propagation.
2019/02/28 15:15:57 [INFO] [foo.example-b.org] acme: Waiting for DNS record propagation.
[...]
2019/02/28 15:15:59 [INFO] [foo.example-b.org] acme: Waiting for DNS record propagation.
2019/02/28 15:17:54 [INFO] [foo.example-b.org] acme: Cleaning DNS-01 challenge
2019/02/28 15:17:55 http: TLS handshake error from 10.255.0.2:35810: [foo.example-b.org] failed to obtain certificate: acme: Error -> One or more domains had a problem:
[foo.example-b.org] time limit exceeded: last error: NS erin.ns.cloudflare.com. did not return the expected TXT record [fqdn: example-a.com., value: ****]: ca3-****
2019/02/28 15:17:55 http: TLS handshake error from 10.255.0.2:35808: no certificate available for 'foo.example-b.org'
2019/02/28 15:23:29 http: TLS handshake error from 10.255.0.2:40520: hostname '' does not qualify for certificate

So it actually picks up on example-a.com (see third line from bottom)?

---

[EDIT]

Oh, I forgot. While the request is waiting to get validated I see the txt dns record in the Cloudflare DNS dashboard for domain example-b.org.

[riker09]

I did more testing and when I remove the CNAME record from example-b.org that points to example-a.com and enter the IP address as A record, everything works fine. So this seems like a bug in the cloudflare dns plugin.

[mholt]

@riker09 Now that you mention it, you are probably right (not in the plugin itself exactly, but in the cloudflare provider in lego which has the actual logic) -- I think I've heard of this before, but I haven't looked into the issue, but I know that CNAME can be problematic for some reason. Maybe best to continue this in the lego package.