Caddy: v2: Issues when reloading config while using a custom tls certificate

[paulschwoerer]

Hey!

I ran into an issue using the docker image of caddy v2. I'm pushing auto-generated configuration (based on started/stopped containers) using the Admin API /load endpoint. The call looks as follows:

curl -X POST "http://caddy:2019/load" \
  -H "Content-Type: text/caddyfile" \
  --data-binary "... caddyfile config ..."

Initially, the configuration looks something like (one container is up):

{
    admin 0.0.0.0:2019
    acme_ca "https://acme-staging-v02.api.letsencrypt.org/directory"
    email [email protected]
}
(tls) {
    tls /etc/ssl/dev.crt /etc/ssl/dev.key
}
(compression) {
    encode zstd gzip
}
host.name {
    import tls
    import compression
    reverse_proxy 172.18.0.3:80
}

Now everything works as expected. Caddy uses the procided TLS cert & key and turns off automatic TLS.

After stopping the running container, the following config will be pushed.

{
    admin 0.0.0.0:2019
    acme_ca "https://acme-staging-v02.api.letsencrypt.org/directory"
    email [email protected]
}
(tls) {
    tls /etc/ssl/dev.crt /etc/ssl/dev.key
}
(compression) {
    encode zstd gzip
}

As expected, the host block disappeared.

When starting the container, something odd happens. The config that's getting pushed is equivalent to the first config. However, caddy does not seem to load the custom TLS cert. Therefore, automatic https gets activated. I noticed, the block specifying which TLS certificate to load is missing from the json config (curl caddy:2019/config/).

// present in initial config, but missing in config after stopping and
// starting container
"certificates": {
        "load_files": [
          {
            "certificate": "/etc/ssl/dev.crt",
            "key": "/etc/ssl/dev.key",
            "tags": [
              "cert0"
            ]
          }
        ]
      }

I attached the full json configs after each operation.
after-starting.txt
after-stopping.txt
initial.txt

Let me know if anything is unclear.

Thanks for your work!

[mholt]

Thanks for the report - what happens if your curl command sets -H "Cache-Control: must-revalidate"?

[sarge]

[edit]
Actually these changes look to be there.
I believe this was fixed in https://github.com/sarge/caddy/tree/reload_certificate_files, but unintentionally reverted

[mholt]

@paulschwoerer Can you also try using the latest from the v2 branch?

[paulschwoerer]

@mholt Using Cache-Control does not change the outcome.

Can you also try using the latest from the v2 branch?

That would be v2.0.0-beta.15, correct?

I'd use the following arg in the docker file
ARG CADDY_SOURCE_VERSION=v2.0.0-beta.15

[mholt]

@paulschwoerer Thanks -- no, the latest from this branch, currently commit cfe85a9fe625fea55dc4f809fd91b5c061064508. You'll get all fixes listed here: https://github.com/caddyserver/caddy/compare/v2.0.0-beta.15...v2

[paulschwoerer]

@mholt Using the v2 branch I'm getting the following output in the caddy containers logs when using the /config/ endpoint to push the first configuration I posted above

2020/03/12 21:56:26.992 INFO    serving initial configuration
2020/03/12 21:56:26 [INFO][cache:0xc000211b80] Started certificate maintenance routine
2020/03/12 21:56:27.677 INFO    admin.api   received request    {"method": "POST", "uri": "/load", "remote_addr": "172.21.0.3:45678", "headers": {"Accept":["*/*"],"Cache-Control":["must-revalidate"],"Content-Length":["311"],"Content-Type":["text/caddyfile"],"User-Agent":["curl/7.61.1"]}}
2020/03/12 21:56:27 http: panic serving 172.21.0.3:45678: interface conversion: caddytls.ACMEIssuer is not certmagic.Issuer: missing method Issue
goroutine 23 [running]:
net/http.(*conn).serve.func1(0xc0000b6280)
    net/http/server.go:1772 +0x139
panic(0xfcbc00, 0xc00048b8c0)
    runtime/panic.go:973 +0x396
github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile.ServerType.Setup(0xc0004248a0, 0x2, 0x2, 0xc000435110, 0xc000424840, 0x3, 0x4, 0x0, 0xc000435530, 0x0)
    github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile/httptype.go:190 +0x31e2
github.com/caddyserver/caddy/v2/caddyconfig/caddyfile.Adapter.Adapt(0x132db20, 0x1b76ee0, 0xc000426000, 0x137, 0x200, 0xc000435110, 0x0, 0x0, 0x0, 0x2, ...)
    github.com/caddyserver/caddy/v2/caddyconfig/caddyfile/adapter.go:49 +0x139
github.com/caddyserver/caddy/v2.handleLoad(0x133d980, 0xc000170000, 0xc000192a00, 0x0, 0x0)
    github.com/caddyserver/caddy/v2/admin.go:457 +0x965
github.com/caddyserver/caddy/v2.AdminHandlerFunc.ServeHTTP(0x1191060, 0x133d980, 0xc000170000, 0xc000192a00, 0x10, 0x0)
    github.com/caddyserver/caddy/v2/admin.go:819 +0x44
github.com/caddyserver/caddy/v2.AdminConfig.newAdminHandler.func1.1(0x133d980, 0xc000170000, 0xc000192a00)
    github.com/caddyserver/caddy/v2/admin.go:111 +0x6d
net/http.HandlerFunc.ServeHTTP(0xc000357d00, 0x133d980, 0xc000170000, 0xc000192a00)
    net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc000357c80, 0x133d980, 0xc000170000, 0xc000192a00)
    net/http/server.go:2387 +0x1a5
github.com/caddyserver/caddy/v2.adminHandler.serveHTTP(0x0, 0xc000333110, 0x1, 0x1, 0xc000357c80, 0x133d980, 0xc000170000, 0xc000192a00)
    github.com/caddyserver/caddy/v2/admin.go:305 +0x71
github.com/caddyserver/caddy/v2.adminHandler.ServeHTTP(0x0, 0xc000333110, 0x1, 0x1, 0xc000357c80, 0x133d980, 0xc000170000, 0xc000192a00)
    github.com/caddyserver/caddy/v2/admin.go:273 +0x4f6
net/http.serverHandler.ServeHTTP(0xc0001bc700, 0x133d980, 0xc000170000, 0xc000192a00)
    net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0000b6280, 0x1340cc0, 0xc0000aa1c0)
    net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
    net/http/server.go:2933 +0x35c

[mholt]

Ah... right... hang tight, it's already fixed on a branch...

[mholt]

Gah... I can't push for some reason!

Sorry, but you'll have to wait on a fix. I lost permissions somehow and cannot go any further at this time...

[paulschwoerer]

Ah bummer, let's get back to this tomorrow :)

[mholt]

@paulschwoerer Sorry for the delay! Got things worked out.

Here, this branch should work better -- but it has a lot of other changes, that I'm still kind of testing, so let me know how it works for you! https://github.com/caddyserver/caddy/pull/3125 (commit 06d9607a85c3216b36bcb0a58b3ed504c598abfe)

[paulschwoerer]

Now getting an error during the docker build

Step 7/56 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build -trimpath -tags netgo -ldflags '-extldflags "-static" -s -w' -o /usr/bin/caddy
 ---> Running in da6f73a944a6
go: github.com/caddyserver/[email protected]: parsing /certmagic/go.mod: open /certmagic/go.mod: no such file or directory
ERROR: Service 'caddy' failed to build: The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build -trimpath -tags netgo -ldflags '-extldflags "-static" -s -w' -o /usr/bin/caddy' returned a non-zero code: 1

I'm using this as my Dockerfile for building caddy.

[mholt]

@paulschwoerer Thanks, after investigating it seems that our CI tests have disappeared!

What happened is somehow one line did not get committed -- just my luck, of course. So it built locally, just didn't make its way to the server. Please try again with the latest commit on the pki branch. :) Thank you!

... I will need to find out what happened to our tests... 🤔

[paulschwoerer]

I can confirm it's working perfectly using the pki branch :)
Thanks for your help!

Can you five me an ETA on when this will be available as a docker image release?

[mholt]

Great! Thanks for confirming!!

/cc @hairyhenderson for the Docker question -- I hope to tag beta 16 today or tomorrow or Monday, so probably soon?

[hairyhenderson]

@mholt thanks for the heads-up - I'll keep my ear close to the ground 😉

[paulschwoerer]

Thanks guys!