Caddy: Caddy should not attempt renewal of non-existent domains

[golf1052]

This is really my fault as I forgot to remove my domain configuration after my domain name expired but this caused me some issues.

Caddy will retry obtaining certs for non-existent domains until it hits the Let's Encrypt rate limit which can affect other sites renewing their certs.

Here is an example log (this is the first time it looks like Caddy ran into an issue)

2017/04/15 18:44:05 [INFO] Scanning for expiring certificates
2017/04/15 18:44:05 [INFO] Certificate for [hotlinering.com] expires in 614h55m54.234020861s; attempting renewal
2017/04/15 18:44:05 [INFO][hotlinering.com] acme: Trying renewal with 614 hours remaining
2017/04/15 18:44:06 [INFO][hotlinering.com] acme: Obtaining bundled SAN certificate
2017/04/15 18:44:06 [INFO][hotlinering.com] acme: Could not find solver for: tls-sni-01
2017/04/15 18:44:06 [INFO][hotlinering.com] acme: Trying to solve HTTP-01
2017/04/15 18:44:17 [INFO][hotlinering.com] acme: Trying renewal with 614 hours remaining
2017/04/15 18:44:17 [INFO][hotlinering.com] acme: Obtaining bundled SAN certificate
2017/04/15 18:44:17 [INFO][hotlinering.com] acme: Could not find solver for: dns-01
2017/04/15 18:44:17 [INFO][hotlinering.com] acme: Trying to solve HTTP-01
2017/04/15 18:44:29 [ERROR] too many renewal attempts; last error: acme: Error 400 - urn:acme:error:connection - DNS problem: NXDOMAIN looking up A for hotlinering.com
Error Detail:
        Validation for hotlinering.com:80
        Resolved to:

        Used:

2017/04/15 18:44:29 [INFO] Certificate for [www.golf1052.com ] expires in 638h55m30.776779966s; attempting renewal
2017/04/15 18:44:29 [INFO][www.golf1052.com] acme: Trying renewal with 2146 hours remaining
2017/04/15 18:44:29 [INFO][www.golf1052.com] acme: Obtaining bundled SAN certificate
2017/04/15 18:44:29 [INFO][www.golf1052.com] acme: Trying to solve HTTP-01
2017/04/15 18:44:29 [INFO][www.golf1052.com] The server validated our request
2017/04/15 18:44:29 [INFO][www.golf1052.com] acme: Validations succeeded; requesting certificates
2017/04/15 18:44:39 [INFO][www.golf1052.com] acme: Trying renewal with 2146 hours remaining
2017/04/15 18:44:39 [INFO][www.golf1052.com] acme: Obtaining bundled SAN certificate
2017/04/15 18:44:40 [INFO][www.golf1052.com] acme: Trying to solve HTTP-01
2017/04/15 18:44:40 [INFO][www.golf1052.com] The server validated our request
2017/04/15 18:44:40 [INFO][www.golf1052.com] acme: Validations succeeded; requesting certificates
2017/04/15 18:44:50 [ERROR] too many renewal attempts; last error: acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: too many certificates already issued for exact set of domains: www.golf1052.com
2017/04/15 18:44:50 [INFO] Done checking certificates

What is interesting is that the domain name expired 1/23/17 but it took until 4/15 for Caddy to run into issues, I assume because this was the first time it needed to renew the cert for hotlinering.com.

It would be great if Caddy didn't keep attempting to refresh a cert for a domain name that didn't exist until it hits a rate limit.

[francislavoie]

Maybe we should do a LookupHost to check that the host actually has addresses before starting ACME? That might not be a good idea though because DNS resolution for LE might not match DNS on Caddy's server.

The other option is watching for the DNS problem: NXDOMAIN error and just giving up if that happens.

[mholt]

Thanks for opening an issue, @golf1052 - but why did you not follow the issue template? ☹️ We need all that information to help you...

Which domain expired exactly, the hotlinering one? And what are the _full_ logs (yes, over a long period of time - whatever you have)? You can post a link.

@francislavoie We discussed doing both of those things you suggested in lego/acme, but decided against it for various reasons, one of which you mentioned. The correct answer to this problem is to not let your domains expire. But I want to dig deep here with @golf1052 first and make sure there's not a bug in Caddy. But we'll need more information.

The rate limit being hit isn't directly because another domain expired. It's because apparently LE issued too many of that certificate in the last 7 days. (Have you been running Caddy over and over again or some other ACME client?)

[golf1052]

Sorry about that I thought this is more of a feature than a bug since it was a user error bug 😅

1. What version of Caddy are you using (caddy -version)?

On 0.9 when upgrading from 0.8.2

2. What are you trying to do?

Start caddy

3. What is your entire Caddyfile?

https://gist.github.com/golf1052/16f6a1b5141887ee3274

4. How did you run Caddy (give the full command and describe the execution environment)?

./caddy -agree -conf="./golf1052.conf" -email="[email protected]" -log="/var/log/caddy/caddy.log" -pidfile="./caddy.pid"

5. Please paste any relevant HTTP request(s) here.

N/A

6. What did you expect to see?

Caddy start up

7. What did you see instead (give full error messages and/or log)?

Don't have the exact error from that command but the full caddy.log
https://gist.github.com/golf1052/1950b3d4002aac1f782bdaaf79dba2ac

8. How can someone who is starting from scratch reproduce the bug as minimally as possible?

1. Register two domain names. Example: areallylongnotowneddomainname.com
2. Setup caddy with automatic https on both domain names.

Caddyfile

areallylongnotowneddomainname.com
anotherreallylongnotowneddomainname.com

1. Start caddy
2. Unregister one of the domain names
3. Wait until caddy needs to renew the cert for the unregistered domain name.
4. Stop caddy
5. Remove the unregistered domain name from the Caddyfile.
6. Attempt to restart caddy
7. Caddy will not start due to hitting the rate limit while trying to renew the cert for the unregistered domain.

[mholt]

Thanks @golf1052 ! That's quite helpful. Is the log that you provided above the _full_ log? It will help a lot, I need to look for certain things in the history of Caddy's execution.

[golf1052]

Yep that's my full caddy log, dating back to when I first switched to caddy. Thanks!

[mholt]

There was absolutely nothing else in the log for 3 months?

[golf1052]

I'm confused. The log for me ends at 2017/04/26. Here's the raw link https://gist.githubusercontent.com/golf1052/1950b3d4002aac1f782bdaaf79dba2ac/raw/e605a479842f70bcf35fca1778773803f491a6d0/caddy.log

[mholt]

Ah! Thanks, that full log is precisely what I needed.

Exactly when did you upgrade to 0.9, and are you still running it? (That version is almost a year old...)

[golf1052]

I am currently running 0.10

I switched from Caddy 0.8.2 to 0.9 on 2017/04/24 at around 04:00. I had also changed the Caddyfile (you can see the latest revision https://gist.github.com/golf1052/16f6a1b5141887ee3274/revisions).

Initially I attempted to start the server using sudo service caddy start (upstart file here https://gist.github.com/golf1052/9e780bcdc698240463c2) but I noticed the server didn't start so I ran the command manually (under my caddy user) using the command from above and that's when I saw the cert renewal error. This error for some reason is not in the logs. It didn't complain about attempting to renew the cert for a non existing domain name (hotlinering.com) but instead first failed when trying to renew the cert for either golf1052.com or attentionpassengers.com. In order to get Caddy running again I had to turn tls off on all my sites.

On 2017/04/25 at around 19:00 I upgraded from 0.9 to 0.10. Keeping the same Caddyfile with tls turned off. After that I slowly reintroduced tls back to my sites. You should see in the log that www.golf1052.com and golf1052.com got https first, then I turned it back on for blog.golf1052.com, then attentionpassengers.com, then the rest of my sites that had tls originally.

[mholt]

(Ignore any comment notifications you may have gotten from a user account that looked like mine earlier today.)

[mholt]

@golf1052 Based on the logs, it looks you were experiencing bugs that were present in older versions of Caddy (0.8 tree was the first one to pilot automatic HTTPS and it has bugs with it).

I think this is not a problem (anymore). You have a lot of subdomains, best to ease those into HTTPS at the rate of about 15-20 per week so you don't hit LE's rate limits. Sounds like your strategy when you updated to 0.10 is on the right track. :+1:

Thanks for your report, and for finally upgrading your version of Caddy!

[golf1052]

Thanks for the investigation and thanks for Caddy!