Caddy: Caddy produces account rate limit errors when compiled with latest certmagic package

[whalehub]

1. What version of Caddy are you using (caddy -version)?

Working build: Caddy 0.11.1 (+0684cf8 Wed Dec 19 15:48:39 UTC 2018) (unofficial)
Broken build: Caddy 0.11.1 (+0b83014 Thu Dec 20 10:24:39 UTC 2018) (unofficial)

2. What are you trying to do?

I'm trying to run a build of Caddy that has been compiled against the latest commit.

3. What is your entire Caddyfile?

(headers) {

   header / {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-XSS-Protection "1; mode=block"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "sameorigin"
        Referrer-Policy "same-origin"
        Expect-CT "max-age=86400, enforce, report-uri=\"<redacted>""
        Feature-Policy "geolocation 'none'; camera 'none'; microphone 'none'"
        Content-Security-Policy "
          default-src 'none' ;
          script-src 'self' ;
          style-src 'self' ;
          img-src * ;
          font-src 'self' data: ; 
          connect-src 'self' data: ; 
          media-src 'self' data: ; 
          object-src 'none' ; 
          worker-src 'self' ; 
          manifest-src 'self' ; 
          frame-ancestors 'self' ; 
          form-action 'self' ; 
          upgrade-insecure-requests ; 
          block-all-mixed-content ; 
          report-uri <redacted>;"
        -Server
    }
}

(ecc-tls-wildcard) {

   tls {
       protocols tls1.2 tls1.3
       dns gandiv5
       ciphers ECDHE-ECDSA-AES128-GCM-SHA256 TLS13-AES-128-GCM-SHA256 ECDHE-ECDSA-WITH-CHACHA20-POLY1305 TLS13-CHACHA20-POLY1305-SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 TLS13-AES-256-GCM-SHA384
       key_type p384
       must_staple
       wildcard
    }
}

(whitelist) {

   ipfilter / {
       rule allow
       ip <redacted>
    }
}

redacted.domain.tld {

   proxy / <redacted> {
       transparent
       websocket
    }

   import headers
   import ecc-tls-wildcard
   import whitelist
}

4. How did you run Caddy (give the full command and describe the execution environment)?

I'm running Caddy on Debian 9.6 with this command:

caddy -agree -http2 -log /opt/caddy/log -root /opt/caddy/webroot -conf /opt/caddy/Caddyfile

5. Please paste any relevant HTTP request(s) here.

N/A

6. What did you expect to see?

I expect Caddy to either:

a.) use the existing account in my .caddy folder to fetch certificates or
b.) successfully register a new account in case none exist in the .caddy folder.

7. What did you see instead (give full error messages and/or log)?

Caddy ignores any existing accounts in the .caddy folder and will always try to register a new account. When trying to register a new account, Caddy produces a rate limit error despite it being the the first registration attempt within 24 hours (i.e., it's not actually hitting LetsEncrypt's "10 registrations per 3 hours" rate limit).

Here's the output from the log file:

Activating privacy features...

Your sites will be served over HTTPS automatically using Let's Encrypt.
By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
  https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don't recommend it.
  Email address:
2018/12/20 11:36:53 registration error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:rateLimited :: Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/, url:

Note: This problem does not occur when you use a build of Caddy that has been compiled against the second latest commit. That build of Caddy can both use existing accounts in the .caddy folder as well as register a new account in case none are present in the .caddy folder.

8. How can someone who is starting from scratch reproduce the bug as minimally as possible?

1. Install Go.
2. Compile Caddy against the master branch with the following command:

go get github.com/mholt/caddy/caddy && \
go get github.com/caddyserver/builds && \
cd $GOPATH/src/github.com/mholt/caddy/caddy && \
go run build.go

1. Run Caddy with the following command:

caddy -agree -http2 -log /opt/caddy/log -root /opt/caddy/webroot -conf /opt/caddy/Caddyfile

[whalehub]

@mholt What are your thoughts on the issue?

[mholt]

Hm, this is working fine for me using the latest commit.

It doesn't make any sense... the only thing that commit does which is even close to relevant is change FileStorage{} to &FileStorage{}.

How sure are you that the caddy built with the latest commit is using the same $HOME and rest of the environment?

[whalehub]

@mholt Thanks for your feedback. I'm not 100% sure that it's using the same $HOME and other environment variables. What would be the easiest way for me to ensure that they're identical for both builds?

[mholt]

env before running Caddy.

[whalehub]

Ok, I can confirm that $HOME and the other environment variables are identical for both builds. I uploaded my compiled binaries to my Seafile server just now. Would you mind running both of them with the command I provided and posting your results here?

[mholt]

Thanks for the binaries, but I don't use Linux, so I'd have to spin up a VM. That's doable, but I don't think the build itself is the problem. I suspect there's something more intricate that involves the state of your disk, especially the $CADDYPATH (~/.caddy by default), and some inconsistency or bug in the code.

Unreasonable to ask, but a VM that reproduces the issue is most effective here.

Since that's probably difficult, could you help debug it? Putting log.Println statements through select areas of CertMagic would be helpful, especially in storage-related code. Could you start by putting a Print in the Filename function here: https://github.com/mholt/certmagic/blob/a3b276a1b44e1c2c3dcab752729976ea04f4839b/filestorage.go#L120 - this will print all filenames that are being accessed on disk, related to certificate stuff (presumably).

I'd also love to see the path through this function: https://github.com/mholt/certmagic/blob/a3b276a1b44e1c2c3dcab752729976ea04f4839b/user.go#L82 -- which shows the subscriber agreement prompt. If you already have a user from before, it shouldn't reach that branch; so litter that function with prints and find out where it goes.

And then put plenty of Prints where ever else you feel might be useful! Feel free to explore beyond what I've suggested, that will speed things up.

[whalehub]

@mholt Which software are you using for creating VMs? I could create a VM that reproduces the issue with VirtualBox and share the .ova file with you or with VMware Workstation Player and share the .vmdk file with you. Let me know which option you'd prefer.

I'll also try out the steps you suggested to debug the issue and get back to you once I have some results.

[mholt]

It'll _probably_ be easier to just debug on your end, to be honest (my download is 5 Mbps unless I borrow the university connection, haven't had any time for that lately though). And I use Parallels...

[whalehub]

@mholt What do you think about me temporarily giving you SSH access to my server for debug purposes?

[mholt]

I'm on holiday for a couple weeks, so I'm kind of taking it easy on the coding front for a bit.

Edit: Also, I don't do that as a matter of policy, for liability reasons, without a waiver.

[whalehub]

@mholt That's fair. I just took a look at what you wanted me to do for debugging Caddy, but I feel a little bit lost. I have virtually zero programming knowledge so I don't know what you mean when you say that I should put "log.Println statements through select areas of CertMagic." I assume that I'm supposed to fetch the Caddy source code with go get and then make some sort of changes in the filestorage.go and user.go files before compiling Caddy from source.

If you could tell me what exactly I need to insert in those files, it would be really helpful for me.

[whalehub]

@mholt I discovered the cause of this issue. You cannot leave the e-mail address field blank when launching Caddy and you're prompted to enter one (even though it says that you can leave it blank). Once I entered an e-mail, I was able to launch Caddy's latest build successfully.

[mholt]

I see... hmm... can you do tree ~/.caddy and put the output here? If you can figure it, I'd like to see that tree before you did the run where you entered the email address... or at least tell me which email you entered, so I can gauge why that made a difference. To the outside, there should not have been any breaking changes.

[whalehub]

@mholt Here is the output from tree ~/.caddy:

I'm afraid I don't have the tree from when I did the run with a blank e-mail address.

The e-mail address I entered the first time was simply a blank one, meaning that I just hit the Enter key when prompted to enter an e-mail address. The second time around, I entered "[email protected]" as the e-mail address, which allowed Caddy to register a new LetsEncrypt account.. Afterwards, I deleted the .caddy folder and started Caddy with the command line argument -email [email protected]. This also allowed Caddy to register a new account.

[mholt]

Cool, thanks. Running with a blank email address should cause the subdirectory within users to be called default, so I'm surprised that that folder doesn't exist -- it should _still_ exist from before, if in fact it was creating that directory before.

So, it seems that Caddy/CertMagic was correctly asking for an email address, since it didn't find an existing user account given a blank email address.

There might still be a bug here, but I'm not 100% sure. Are you able to reproduce the behavior more concretely? To try, set the CADDYPATH env variable to some value other than $HOME/.caddy and use the -ca command line flag to use Let's Encrypt's (v2) staging endpoint: https://letsencrypt.org/docs/staging-environment/

[whalehub]

@mholt I was unable to reproduce this behavior again.

[crvv]

I encountered this today.

The rate limit is

You can create a maximum of 10 Accounts per IP Address per 3 hours

https://letsencrypt.org/docs/rate-limits/

In log, there are 10 lines of Email address: 2019/01/17 08:32:39 [INFO][domain name] Obtain: Certificate already exists in storage, and it failed at the 11th times.

log:

Jan 17 08:26:49 caddy[29126]: Activating privacy features...
Jan 17 08:26:49 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:49 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:49 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:49 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:49 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:49 caddy[29126]:   Email address: 2019/01/17 08:26:46 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:49 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:49 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:49 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:49 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:49 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:49 caddy[29126]:   Email address: 2019/01/17 08:26:47 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:49 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:49 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:49 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:49 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:49 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:49 caddy[29126]:   Email address: 2019/01/17 08:26:48 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:51 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:51 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:51 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:51 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:51 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:51 caddy[29126]:   Email address: 2019/01/17 08:26:49 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:51 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:51 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:51 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:51 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:51 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:51 caddy[29126]:   Email address: 2019/01/17 08:26:50 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:51 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:51 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:51 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:51 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:51 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:54 caddy[29126]:   Email address: 2019/01/17 08:26:51 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:54 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:54 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:54 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:54 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:54 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:54 caddy[29126]:   Email address: 2019/01/17 08:26:52 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:54 caddy[29126]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:26:54 caddy[29126]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:26:54 caddy[29126]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:26:54 caddy[29126]: Please enter your email address to signify agreement and to be notified
Jan 17 08:26:54 caddy[29126]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:26:54 caddy[29126]:   Email address: 2019/01/17 08:26:53 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:26:55 caddy[29126]: done.
Jan 17 08:32:36 caddy[29126]: 2019/01/17 08:32:36 [INFO] SIGTERM: Shutting down servers then terminating
Jan 17 08:32:36 caddy[29126]: 2019/01/17 08:32:36 [INFO][FileStorage:/var/lib/caddy] Stopped certificate maintenance routine


Jan 17 08:32:37 caddy[29485]: Activating privacy features...
Jan 17 08:32:37 caddy[29485]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:32:37 caddy[29485]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:32:37 caddy[29485]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:32:37 caddy[29485]: Please enter your email address to signify agreement and to be notified
Jan 17 08:32:37 caddy[29485]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:32:38 caddy[29485]:   Email address: 2019/01/17 08:32:38 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:32:38 caddy[29485]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:32:38 caddy[29485]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:32:38 caddy[29485]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:32:38 caddy[29485]: Please enter your email address to signify agreement and to be notified
Jan 17 08:32:38 caddy[29485]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:32:39 caddy[29485]:   Email address: 2019/01/17 08:32:39 [INFO][domain name] Obtain: Certificate already exists in storage
Jan 17 08:32:39 caddy[29485]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:32:39 caddy[29485]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:32:39 caddy[29485]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:32:39 caddy[29485]: Please enter your email address to signify agreement and to be notified
Jan 17 08:32:39 caddy[29485]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:32:40 caddy[29485]:   Email address: 2019/01/17 08:32:40 registration error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:rateLimited :: Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/, url:


Jan 17 08:33:05 caddy[29560]: Activating privacy features...
Jan 17 08:33:05 caddy[29560]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Jan 17 08:33:05 caddy[29560]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Jan 17 08:33:05 caddy[29560]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Jan 17 08:33:05 caddy[29560]: Please enter your email address to signify agreement and to be notified
Jan 17 08:33:05 caddy[29560]: in case of issues. You can leave it blank, but we don't recommend it.
Jan 17 08:33:05 caddy[29560]:   Email address: 2019/01/17 08:33:05 registration error: acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:rateLimited :: Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/, url:

[francislavoie]

Looks like it might be the same thing as #2432?

Edit: haha scratch that, you were the one to open that issue. I was reading from emails and it wasn't obvious it was the same person.

[mholt]

Do you have a reproducible test case?

[crvv]

There are three bugs around this issue.

1. In caddy 0.11.1
   https://github.com/mholt/caddy/blob/v0.11.1/caddytls/config.go#L207
   ObtainCert will check if the certificate already exists.
   But in caddy 0.11.2, it doesn't check.
   https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/config.go#L272

2. ObtainCert calls preObtainOrRenewChecks
   https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/config.go#L352
   And calls getEmail
   https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/user.go#L82.
   If the email is "", it will create a new user
   https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/user.go#L101
   and save it to disk. The previous user is overrided.
   https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/user.go#L141
   Then newACMEClient in ObtainCert will read the file again, get a empty user, and register a new Let's Encrypt account.

3. email in Caddyfile doesn't work, which is fixed by #2432

2 only occurs if the email is empty, so #2432 will get rid of the bug if there is an email in Caddyfile.

[mholt]

Great, thanks for the writeup. I'm looking at this now.

[mholt]

Okay, I'm pushing a fix for 1 in just a moment.

As for 2, I'm not quite sure it works like you've described. Everything is right until "The previous user is overrided" -- there shouldn't be any previous user (with the same email address, including an empty email), since it would have loaded it from disk. Then newACMEClient in ObtainCert will read the file again, get a _non-empty user (even if the email address is empty)_, and _will use the existing CA account_: https://github.com/mholt/caddy/blob/v0.11.2/vendor/github.com/mholt/certmagic/user.go#L159-L163

In other words, I think that fixing 1 and 3 is sufficient. I will comment in a few minutes when I have pushed my changes. I hope you can help test them!

[mholt]

Interesting, didn't know I could close issues across repositories like this.

Anyway @crvv I've pushed the fix in f3a4f46. Please have a look and then do a quick test if you could, to be a second verification that it is working as intended. :) Thank you!

[crvv]

Everything is right until "The previous user is overrided" -- there shouldn't be any previous user (with the same email address, including an empty email), since it would have loaded it from disk.

getEmail doesn't check whether the user exists. It only checks whether the email is "".
If the email is "", it will create a new user and won't load previous user from disk.

Please see mholt/certmagic#17 for details.

[mholt]

Thanks for that PR, @crvv. After spending more time investigating, I can confirm you are right. Merged it in, and will update Caddy soon.

[UniverseXXX]

Thanks for that PR, @crvv. After spending more time investigating, I can confirm you are right. Merged it in, and will update Caddy soon.

Hi @mholt Are you planing to release an updated version any time soon? thanks.

[mholt]

Yes very soon, we just need to be able to explain #2407 and finish https://github.com/mholt/certmagic/pull/23.

[mholt]

@crvv @whalehub @UniverseXXX Following up on related issues, could you help test https://github.com/mholt/caddy/pull/2452 please? Need to make sure the change is solid before doing a release (but I won't wait long, either).

[UniverseXXX]

@mholt Sorry couldn't get back to you earlier. Just downloaded 0.11.3 and it doesn't work, unfortunately. Here is the log:

Feb  5 21:44:07 RPi systemd[1]: caddy.service: Failed with result 'exit-code'.
Feb  5 21:44:07 RPi systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Feb  5 21:44:07 RPi caddy[1193]: Activating privacy features... get Agreement URL: Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on [::1]:53: read udp [::1]:34351->[::1]:53: read: connection refused
Feb  5 21:44:01 RPi systemd[1]: Started Caddy HTTP/2 web server.
Feb  5 22:01:36 RPi systemd[1]: Stopped Caddy HTTP/2 web server.
Feb  5 22:01:36 RPi systemd[1]: Stopping Caddy HTTP/2 web server...

[mholt]

It works. Looks like your network is having trouble connecting to Let's Encrypt.

[UniverseXXX]

It works. Looks like your network is having trouble connecting to Let's Encrypt.

@mholt You are right. Sorry, my bad - got Pi-Hole running in Docker which causing the issue.
Got some messages in the log, however, it's working fine in the end. Thanks a lot.

Feb  5 22:29:39 RPi caddy[16296]:   Email address: done.
Feb  5 22:29:39 RPi caddy[16296]: in case of issues. You can leave it blank, but we don't recommend it.
Feb  5 22:29:39 RPi caddy[16296]: Please enter your email address to signify agreement and to be notified
Feb  5 22:29:39 RPi caddy[16296]:   https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Feb  5 22:29:39 RPi caddy[16296]: By continuing, you agree to the Let's Encrypt Subscriber Agreement at:
Feb  5 22:29:39 RPi caddy[16296]: Your sites will be served over HTTPS automatically using Let's Encrypt.
Feb  5 22:29:39 RPi caddy[16296]:   Email address:

[mholt]

Yeah I recommend running with the -email and -agree CLI flags so you don't get prompted. See the docs: https://caddyserver.com/docs/cli