Caddy: Caddy still trying to hit let's encrypt even with local_certs enabled

[ggilmore]

1. My Caddy version (caddy version):

v2.0.0-beta.18 h1:N8CR8nO+n9Sk16CBx6ysVZWIZv0hgtU1J84st1bFWgI=

2. How I run Caddy:

a. System environment:

macOS Catalina v 10.15.3

/etc/hosts:

   1   │ ##
   2   │ # Host Database
   3   │ #
   4   │ # localhost is used to configure the loopback interface
   5   │ # when the system is booting.  Do not change this entry.
   6   │ ##
   7   │ 127.0.0.1        localhost sourcegraph.test
   8   │ 255.255.255.255  broadcasthost
   9   │ ::1              localhost
  10   │ # Added by Docker Desktop
  11   │ # To allow the same kube context to work on the host and the container:
  12   │ 127.0.0.1        kubernetes.docker.internal
  13   │ # End of section
  14   │ # Added by Geoffrey
  15   │ 127.0.0.1        ggilmore.nice

b. Command:

env SOURCEGRAPH_HTTPS_DOMAIN=sourcegraph.test SOURCEGRAPH_HTTPS_PORT=3443 caddy run --config=dev/Caddyfile

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

{
    local_certs
}


https://{$SOURCEGRAPH_HTTPS_DOMAIN}:{$SOURCEGRAPH_HTTPS_PORT} {
    reverse_proxy localhost:3080
    tls internal
}

3. The problem I'm having:

I'm trying to use caddy as a suitable https proxy for local development of Sourcegraph. We have the app serving over http at localhost:3080, and we'd like to have caddy reverse proxy that over https://sourcegraph.test:3443 (sourcegraph.test is a fake domain that we put in our own /etc/hosts for local testing purposes).

When I try to run Caddy with this configuration, I get this error message:

4. Error messages and/or full log output:

2020/03/20 19:43:35.113 INFO    using provided configuration    {"config_file": "dev/Caddyfile", "config_adapter": ""}
2020/03/20 19:43:35.115 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/03/20 12:43:35 [INFO][cache:0xc000953e00] Started certificate maintenance routine
2020/03/20 19:43:35.122 INFO    http    enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2020/03/20 19:43:35.232 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2020/03/20 19:43:35.233 INFO    tls cleaned up storage units
2020/03/20 19:43:35.233 INFO    http    enabling automatic TLS certificate management   {"domains": ["sourcegraph.test"]}
2020/03/20 19:43:35.234 INFO    autosaved config    {"file": "/Users/ggilmore/Library/Application Support/Caddy/autosave.json"}
2020/03/20 19:43:35.234 INFO    serving initial configuration
2020/03/20 12:43:35 [INFO][sourcegraph.test] Obtain certificate; acquiring lock...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Obtain: Lock acquired; proceeding...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Waiting on rate limiter...
2020/03/20 12:43:35 [INFO][sourcegraph.test] Done waiting
2020/03/20 12:43:35 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:43:35 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=tls-alpn-01 remaining=[http-01])
2020/03/20 12:43:37 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:43:37 [ERROR] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=http-01 remaining=[])
2020/03/20 12:43:39 [ERROR] attempt 1: [sourcegraph.test] Obtain: [sourcegraph.test] acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  - retrying in 1m0s (4.374944404s/720h0m0s elapsed)...
2020/03/20 12:43:48 http: TLS handshake error from 127.0.0.1:55664: no certificate available for 'sourcegraph.test'
2020/03/20 12:43:48 http: TLS handshake error from 127.0.0.1:55665: no certificate available for 'sourcegraph.test'
2020/03/20 12:44:39 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:44:40 [ERROR] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=tls-alpn-01 remaining=[http-01])
2020/03/20 12:44:42 [INFO] [sourcegraph.test] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:44:42 [ERROR] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  (challenge=http-01 remaining=[])
2020/03/20 12:44:44 [ERROR] attempt 2: [sourcegraph.test] Obtain: [sourcegraph.test] acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rejectedIdentifier :: Error creating new order :: Cannot issue for "sourcegraph.test": Domain name does not end with a valid public suffix (TLD), url:  - retrying in 2m0s (1m9.040441757s/720h0m0s elapsed)...
^[[A^[[A^C2020/03/20 19:46:42.220   INFO    shutting down   {"signal": "SIGINT"}
2020/03/20 12:46:42 http: TLS handshake error from 127.0.0.1:55666: read tcp 127.0.0.1:3443->127.0.0.1:55666: use of closed network connection
2020/03/20 12:46:42 [INFO][sourcegraph.test] Obtain: Releasing lock
2020/03/20 12:46:42 [INFO][cache:0xc000953e00] Stopped certificate maintenance routine
2020/03/20 19:46:42.221 INFO    shutdown done   {"signal": "SIGINT"}

What's strange is that Caddy still seems to be trying to contact Lets Encrypt even though I enabled local_certs in the global options.

Both Chrome and Firefox complain about SSL errors whenever I try to hit https://sourcegraph.test:3443

5. What I already tried:

I've tried changing the fake domain to sourcegraph.dev (a real TLD), but I still get SSL errors and Caddy still seems to be trying to hit Let's Encrypt even though I enabled local_certs in the global options.

Caddy Logs

2020/03/20 19:47:23.655 INFO    using provided configuration    {"config_file": "dev/Caddyfile", "config_adapter": ""}
2020/03/20 19:47:23.657 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/03/20 12:47:23 [INFO][cache:0xc000477770] Started certificate maintenance routine
2020/03/20 19:47:23.664 INFO    http    enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2020/03/20 19:47:23.776 INFO    pki.ca.local    root certificate is already trusted by system   {"path": "storage:pki/authorities/local/root.crt"}
2020/03/20 19:47:23.777 INFO    tls cleaned up storage units
2020/03/20 19:47:23.777 INFO    http    enabling automatic TLS certificate management   {"domains": ["sourcegraph.dev"]}
2020/03/20 19:47:23.777 INFO    autosaved config    {"file": "/Users/ggilmore/Library/Application Support/Caddy/autosave.json"}
2020/03/20 19:47:23.777 INFO    serving initial configuration
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Obtain certificate; acquiring lock...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Obtain: Lock acquired; proceeding...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Waiting on rate limiter...
2020/03/20 12:47:23 [INFO][sourcegraph.dev] Done waiting
2020/03/20 12:47:23 [INFO] [sourcegraph.dev] acme: Obtaining bundled SAN certificate given a CSR
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3470502029
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: Could not find solver for: tls-alpn-01
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: use http-01 solver
2020/03/20 12:47:24 [INFO] [sourcegraph.dev] acme: Trying to solve HTTP-01

6. Links to relevant resources:

Here is the PR that I'm working on: https://github.com/sourcegraph/sourcegraph/pull/9050

_Moved from https://caddy.community/t/caddy-still-trying-to-hit-lets-encrypt-even-with-local-certs-enabled/7268_

[mholt]

@ggilmore Thank you for the detailed report! I was able to reproduce the issue quite easily with that information.

I spent most of the evening working on a fix, which you can find in d692d50. That commit should work much better for you. I'll also leave a comment in your upstream PR with a tip about your Caddyfile, so check it out!

Let me know how it goes!

[mholt]

@ggilmore Just so you know, if you can test it tomorrow (Monday) to verify that it works I will tag beta 19 and release it tomorrow. :+1:

[ggilmore]

Ah, sorry that I missed your message before I signed off for the weekend.
Thanks for being responsive! I’ll test it within the next three hours and
report back to you

On Sun, Mar 22, 2020 at 5:55 PM Matt Holt [e-mail] wrote:

@ggilmore https://github.com/ggilmore Just so you know, if you can test
it tomorrow (Monday) to verify that it works I will tag beta 19 and release
it tomorrow. 👍

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/caddyserver/caddy/issues/3164#issuecomment-602310154,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACE2UO54KAKD44SJHH4KGDTRI2XO5ANCNFSM4LQVOS4A
.

[mholt]

@ggilmore No worries, that's the point of a weekend, especially in times like these :) I've just felt a little cooped up so my mind's been going 100mph and so it was an overly-productive weekend for me... heh.

If it helps, here are some recent builds from CI: https://github.com/caddyserver/caddy/actions/runs/61533439

[ggilmore]

Hey @mholt I manually built caddy from https://github.com/caddyserver/caddy/commit/348cb798e26a056bd40f5ec7c0d8f440d7b6cc0b and everything seems to work now with the following Caddyfile and logs

{$SOURCEGRAPH_HTTPS_DOMAIN}:{$SOURCEGRAPH_HTTPS_PORT}
reverse_proxy localhost:3080
tls internal

10:13:10               caddy | {"level":"info","ts":1584983590.740164,"msg":"using provided configuration","config_file":"dev/Caddyfile","config_adapter":""}
10:13:10               caddy | {"level":"info","ts":1584983590.745651,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["localhost:2019"]}
10:13:10               caddy | 2020/03/23 10:13:10 [INFO][cache:0xc0006ab9a0] Started certificate maintenance routine
10:13:10               caddy | {"level":"info","ts":1584983590.757461,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
10:13:11               caddy | {"level":"info","ts":1584983591.203548,"logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
10:13:11               caddy | {"level":"info","ts":1584983591.2070951,"logger":"tls","msg":"cleaned up storage units"}
10:13:11               caddy | {"level":"info","ts":1584983591.207257,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sourcegraph.test"]}
10:13:11               caddy | 2020/03/23 10:13:11 [WARNING] Stapling OCSP: no OCSP stapling for [sourcegraph.test]: no OCSP server specified in certificate
10:13:11               caddy | {"level":"info","ts":1584983591.208641,"msg":"autosaved config","file":"/Users/ggilmore/Library/Application Support/Caddy/autosave.json"}
10:13:11               caddy | {"level":"info","ts":1584983591.208653,"msg":"serving initial configuration"}

Thank you so much for looking into this!

[mholt]

Excellent!! Thanks for confirming! I will tag beta 19 momentarily.