Caddy: v2: if custom TLS certificate is set, Let's Encrypt sites break

[theel0ja]

example.com {
        root * /var/www/example.com/public
        file_server

       tls /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.key
}

cdn.example.com {
       root * /var/www/cdn.example.com/public
       file_server

       tls /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.key
}

example.org {
    respond "example.org"
}

In this case, example.com and cdn.example.com work, but example.org returns error message in browsers (in Firefox, SSL_ERROR_INTERNAL_ERROR_ALERT) and into Caddy's logs.

If I remove the example.com blocks, example.org works.

caddy[1228]: 2020/01/26 02:22:18 http: TLS handshake error from CLIENTIP:CLIENTPORT: no server TLS configuration available for ClientHello: &{CipherSuites:[19018 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:example.org SupportedCurves:[56026 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[43690 772 771 770 769] Conn:0xc0000f2028}

$ caddy version
v2.0.0-beta.13 h1:QL0JAepFvLVtOatABqniuDRQ4HmtvWuuSWZW24qVVtk=

[Henrocker]

Yes, this is exactly the same that happend to me: https://caddy.community/t/v2-automatic-https-certificate-errors/6847/21

[mholt]

I believe I've fixed this in https://github.com/caddyserver/caddy/commit/4a07a5d41e0f54d1a1ec998b9d956ccf2a880d90 -- please pull the latest from the v2 branch and try again, if you would!

[Henrocker]

I've just rebuild Caddy v2 with latest master (including https://github.com/caddyserver/caddy/commit/57c6f22684e74191814a30f9de83a05c11ac4b78) and I get panics again, when 1. using LE sites only in Caddyfile, and 2. in combination with sites with specified certificate paths in Caddyfile.

Go version 1.13.8 and System is: Debian 10 x86_64.

I tried only specifying one hostname with specified certificate path, and latest master of Caddy v2 works fine with it:
Caddyfile:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}

Logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/18 10:09:54.745 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/18 10:09:54.747 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/18 10:09:54.747 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/18 11:09:54 [INFO][cache:0xc00049a000] Started certificate maintenance routine
2020/02/18 10:09:54.766 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "hnrk.io", "server_name": "srv0"}
2020/02/18 10:09:54.766 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "www.hnrk.io", "server_name": "srv0"}
2020/02/18 10:09:54.766 INFO    http    enabling automatic TLS certificate management   {"domains": []}
2020/02/18 10:09:54.779 INFO    tls     cleaned up storage units
2020/02/18 10:09:54.779 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/02/18 10:09:54.779 INFO    serving initial configuration
^C2020/02/18 10:10:14.846       INFO    shutting down   {"signal": "SIGINT"}

When however specifying a single LE site, Caddy breaks:
Caddyfile:

harmoniks.de, www.harmoniks.de {
        tls [email protected]
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/18 10:09:28.182 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/18 10:09:28.186 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/18 10:09:28.186 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/02/18 10:09:28.186 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/18 11:09:28 [INFO][cache:0xc0001fa6e0] Started certificate maintenance routine
panic: reflect: call of reflect.Value.FieldByName on ptr Value

goroutine 1 [running]:
reflect.flag.mustBe(...)
        /root/go1.13.8/go/src/reflect/value.go:208
reflect.Value.FieldByName(0xf7b7c0, 0xc0006a4b78, 0x196, 0x10d708d, 0xd, 0x196, 0x7d445c, 0xc000480380)
        /root/go1.13.8/go/src/reflect/value.go:890 +0x1ed
github.com/caddyserver/caddy/v2.Context.LoadModule(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf02720, 0xc0006a4b78, 0x10d708d, ...)
        /etc/buildv2caddy/v2caddy/context.go:131 +0x102
github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).Provision(0xc0002a6fa0, 0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0x4, 0x4)
        /etc/buildv2caddy/v2caddy/modules/caddytls/tls.go:92 +0xb49
github.com/caddyserver/caddy/v2.Context.LoadModuleByID(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xc00035daf4, 0x3, 0xc0002a43f0, ...)
        /etc/buildv2caddy/v2caddy/context.go:322 +0x66a
github.com/caddyserver/caddy/v2.Context.loadModuleMap(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf00899, 0x0, 0xf8d9c0, ...)
        /etc/buildv2caddy/v2caddy/context.go:261 +0x260
github.com/caddyserver/caddy/v2.Context.loadModulesFromSomeMap(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf00899, 0x0, 0x0, ...)
        /etc/buildv2caddy/v2caddy/context.go:223 +0x107
github.com/caddyserver/caddy/v2.Context.LoadModule(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf0e520, 0xc0006c5b80, 0x10b2a05, ...)
        /etc/buildv2caddy/v2caddy/context.go:198 +0xcce
github.com/caddyserver/caddy/v2.run.func3(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xc0006c5b80, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:387 +0x98
github.com/caddyserver/caddy/v2.run(0xc0006c5b80, 0x501, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:395 +0x28e
github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun(0xc0002eb800, 0x59a, 0x600, 0x7, 0xc000154630)
        /etc/buildv2caddy/v2caddy/caddy.go:249 +0xf7
github.com/caddyserver/caddy/v2.changeConfig(0x10a9066, 0x4, 0x10b27ff, 0x7, 0xc0002ea000, 0x59a, 0x600, 0x423001, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:156 +0x4ce
github.com/caddyserver/caddy/v2.Load(...)
        /etc/buildv2caddy/v2caddy/caddy.go:102
github.com/caddyserver/caddy/v2/cmd.cmdRun(0xc0000ca7e0, 0x0, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/cmd/commandfuncs.go:186 +0x352
github.com/caddyserver/caddy/v2/cmd.Main()
        /etc/buildv2caddy/v2caddy/cmd/main.go:72 +0x23e
main.main()
        /etc/buildv2caddy/v2caddy/cmd/caddy/main.go:37 +0x20

[mholt]

Oops, thanks for the full stack trace @Henrocker that made it quick to find the problem. Fixed, I think, please try again!

[Henrocker]

@mholt Thanks, that's fixed the panic!

I've recompiled new Caddyv2 including latest commit 0b09b07, if I now put a single domain in Caddyfile with specified certificate path, I get the following logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/19 07:19:29.539 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/19 07:19:29.543 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/19 07:19:29.543 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/19 08:19:29 [INFO][cache:0xc000564000] Started certificate maintenance routine
2020/02/19 08:19:29 [INFO][cache:0xc000564000] Stopped certificate maintenance routine
run: loading initial config: loading new config: loading app modules: module name 'tls': provision tls: loading TLS automation management module: module name 'load_files': decoding module config: tls.certificates.load_files: json: cannot unmarshal array into Go value of type caddytls.CertKeyFilePair

Caddyfile:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}

Thank you for your help!

[mholt]

@Henrocker Tricky, tricky... fixed in 0005e3acdc7f0bc89f7a8bb15a1e23295986a3b7, I believe. Sigh. Kind of hacky but that's how it's going to be I guess. I still maintain that the Caddyfile adapter is the hardest part of building Caddy 2.

[Henrocker]

@mholt Thanks, with including 0005e3a, I can now run Caddy v2 using my Caddyfile, however, I think, we're not quite there, yet. 😅

I've tried several config combinations with my Caddyfile and see the following behaviour:

• LE sites work perfectly fine, even in combination with sites with specified certificate in Caddyfile.
• Having One single vhost-block with dedicated specified certificate (and possibly several mutliple LE sites together in Caddyfile) also works fine, BUT:

With two or more vhost blocks that point to dedicated certificate files, Caddy v2 always throws an SSL_Protocol error on the first vhost (so on (www.)hnrk.io, I receive SSL_Protocol errors, but draw.hnrk.io loads just fine). Config looks like this when it happens:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}
draw.hnrk.io {
        root * /etc/caddy/html/draw
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Logs:

./caddyOLD1 run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/20 09:03:48.436 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/20 09:03:48.440 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/20 09:03:48.440 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/20 10:03:48 [INFO][cache:0xc0003f8050] Started certificate maintenance routine
2020/02/20 09:03:48.471 INFO    tls     cleaned up storage units
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "draw.hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "www.hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    enabling automatic TLS certificate management   {"domains": []}
2020/02/20 09:03:48.472 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/02/20 09:03:48.472 INFO    serving initial configuration
2020/02/20 10:04:15 http: TLS handshake error from 62.157.168.126:16859: no certificate available for 'hnrk.io'     <------------ That's the error!
^C2020/02/20 09:04:24.633       INFO    shutting down   {"signal": "SIGINT"}
2020/02/20 10:04:24 [INFO][cache:0xc0003f8050] Stopped certificate maintenance routine
2020/02/20 09:04:24.633 INFO    shutdown done   {"signal": "SIGINT"}

Do you know a way to debug (generate stack traces) out of static go binaries directly on *nix?

Thanks!

[mholt]

@Henrocker What are all the names (exactly) on the certificate at /etc/caddy/hnrk.io.crt?

[Henrocker]

@mholt This is a wildcard certificate.

DNS-Name=*.hnrk.io
DNS-Name=hnrk.io

[mholt]

@Henrocker Hmm, I can't reproduce it.

I generated a certificate just like yours (two SAN names, test-1.lightcodelabs.com and *.lightcodelabs.com) and then used your Caddyfile except loading my certificates and replacing your domains with mine. curl -v --insecure "https://test-1.lightcodelabs.com/"works just fine for all my domains and subdomains that I'm testing with. When I try with some other hostname, like curl -v --insecure "https://localhost/", then I see the no certificate available for 'localhost' error.

In other words, things are working as expected.

I suppose I'll need your help to further debug.

Can you put log.Printf() statements in certmagic/handshake.go (do a search for "no certificate available for") and just kind of see why it can't find your certificate?

Edit: Oooh, I found a way to reproduce it. Still, your help would be greatly appreciated if you can spare a few minutes! :)

[Henrocker]

@mholt I am happy to help, hang on!

[mholt]

Thanks! I just found the problem... might be a little tricky to fix. I'll see about it today if I can.

Somehow I need to prevent cert files that are used repeatedly from being loaded twice:

"certificates": {
                "load_files": [
                    {
                        "certificate": "/Users/matt/Desktop/cert.pem",
                        "key": "/Users/matt/Desktop/key.pem",
                        "tags": [
                            "cert0"
                        ]
                    },
                    {
                        "certificate": "/Users/matt/Desktop/cert.pem",
                        "key": "/Users/matt/Desktop/key.pem",
                        "tags": [
                            "cert1"
                        ]
                    }
                ]
            }

It's looking for cert0, even though cert1 overwrote cert0 in the cache.

[mholt]

@Henrocker Fixed in 99f91c4c6f812ebfae505a8c29a750965af0cfcb -- please test, thanks!

PS. Your use of _actual domain names_ and not redacting any information really made this quick and easy. I hope more people will report bugs the way you do... too many people like to change and hide their configs before posting it, and we can't help them. Seriously it happens every day.

[Henrocker]

@mholt Wow, that was really quick! Thanks, my complete setup works like a charm now.

I really feel the post scriptum part, and thanks to bcrypt for auth, one doesn't even have to look into Caddyfile again before posting it publicly, yay! :-)

[CrazyByDefault]

This issue (or something of the sort) seems to have been reintroduced in 2.1.0 and above. Our Caddyfile which uses both LE and custom certs now starts properly, but dies with

{"level":"info","ts":1595231297.6785583,"msg":"serving initial configuration"}
2020/07/20 07:48:32 http: TLS handshake error from 49.37.128.229:56978: no server TLS configuration available for ClientHello: &{CipherSuites:[64250 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53] ServerName:internal-staging.test.co SupportedCurves:[14906 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537] SupportedProtos:[h2 http/1.1] SupportedVersions:[23130 772 771 770 769] Conn:0xc000188580 config:0xc000609800}
2020/07/20 07:48:32 http: TLS handshake error from 49.37.128.229:56980: no server TLS configuration available for ClientHello: &{CipherSuites:[43690 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53] ServerName:internal-staging.test.co SupportedCurves:[6682 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537] SupportedProtos:[h2 http/1.1] SupportedVersions:[47802 772 771 770 769] Conn:0xc0001885a8 config:0xc000609800}
2020/07/20 07:48:32 http: TLS handshake error from 49.37.128.229:56982: no server TLS configuration available for ClientHello: &{CipherSuites:[14906 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53] ServerName:internal-staging.test.co SupportedCurves:[47802 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537] SupportedProtos:[h2 http/1.1] SupportedVersions:[19018 772 771 770 769] Conn:0xc0001885c0 config:0xc000609800}

after updating to 2.1.0.

Rolling back to 2.0.0 without any changes in config works just fine.

[francislavoie]

@CrazyByDefault I don't think this is the same problem. Could you please open up a new issue? Please include your full Caddyfile. Also did you try 2.1.1 or only 2.1.0?

[CrazyByDefault]

I tried both 2.1.1 and 2.1.0, and only 2.0.0 seems to be working for my config. I'll open a new issue soon, thanks!

[gbhrdt]

I experience exactly the same like @CrazyByDefault . Only 2.0.0 works for us. Did you open a new issue, yet? Otherwise I will do.