Caddy: [Gandi] Unable to obtain certificates HTTPS certificates

[SkYNewZ]

I'm unable to obtain certificates from staging nor production 😞 BUT I can see some TXT records on my domain.
And, the Let's Encrypt URL seems to be frozen at staging environment…

@ 10800 IN SOA ns1.gandi.net. hostmaster.gandi.net. 1602253807 10800 3600 604800 10800
*.cloud 300 IN CNAME cloud.skynewz.dev.
@ 300 IN A 185.199.108.153
@ 300 IN A 185.199.109.153
@ 300 IN A 185.199.110.153
@ 300 IN A 185.199.111.153
@ 10800 IN MX 10 spool.mail.gandi.net.
@ 10800 IN MX 50 fb.mail.gandi.net.
@ 300 IN TXT "v=spf1 include:_mailcust.gandi.net ?all"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "1MXf_ZXP2fsqcL5aoHRp7lEKlFKKfF80nRrqQ_U9KXI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "41fqxzBfjOf60IdI8IwIEX_3re4aEYpJj_1lGBCuu6s"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "41fqxzBfjOf60IdI8IwIEX_3re4aEYpJj_1lGBCuu6s"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "CazgFDn_IrmDw0KXVqs4Kl5-Vv8MKvUwKT_YQzBoz0o"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "KYCFYVONLOBSUoDgM6KZp55POKlZxouvu7WqxR-EfSo"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "VtcQKWe7NBIqe2aBvR5Wy6NBrgP3Q8yIS5yRCmIZ1TI"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "fafuT42wKuL1AjHUOtDuuH0jABkwmdRk1lFl_O9qKmc"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "gswLJvKEqFcuyDgUriNoE_hlgf71USFm4AePzj-NHJ4"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "hVzS2nnXTErK6Xv8D4xGO15q96V2OO5uJT41d4i2Tro"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "slAdKF1G9qWnPknE9Tua1aVg9yux9JWB-ObtBGfx4Uc"
_acme-challenge.cloud.skynewz.dev 10800 IN TXT "tTmFZ_um62V4jNCnirwWB533pq-esRsOCxfDmWvi1As"
cloud 300 IN A 51.15.212.58
…

My caddy log

Oct 09 14:18:54 caddy caddy[5323]: {"level":"debug","ts":1602253134.7423246,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/7775020661","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["98834490"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["799"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 14:18:54 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104mlYmN_nBKomq2y4UJ3wuge1XJfiw6dbLJxbM4dyUnkw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 14:18:54 caddy caddy[5323]: {"level":"error","ts":1602253134.7438054,"logger":"tls.obtain","msg":"will retry","error":"[cloud.skynewz.dev] Obtain: [cloud.skynewz.dev] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/98834490/5601961055) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":123.299836857,"max_duration":2592000}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.3428767,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-staging-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["724"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 14:19:55 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.4860983,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 09 Oct 2020 14:19:55 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002N8f12el4ULLbthKKH9Q4buVJNV5_YX_unIL-S2VOKd0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.6575475,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["16035029"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["361"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 14:19:55 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-staging-v02.api.letsencrypt.org/acme/order/16035029/164345359"],"Replay-Nonce":["0003tLg4KEqPFmM0XoYNwUDMzzROwohwnMLvkxX2Jv12PpE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.8043296,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/129224421","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["16035029"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["816"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 14:19:55 GMT"],"Link":["<https://acme-staging-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0004IM5K7_Ina6typUlSztskb-Kq1SbSoPV4EnEcHe257zU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.8046875,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"debug","ts":1602253195.8047078,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"http-01"}
Oct 09 14:19:55 caddy caddy[5323]: {"level":"info","ts":1602253195.8047197,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}

I tried:

• [x] With a single domain
• [x] With my (orignal) domain + wildcard
• [x] Try to change the acme_ca

Caddyfile

{
    # Enable Debug mode
    debug

       # Disable admin console
    admin off

    # Default email for tls
    email [email protected]

    acme_ca https://acme-v02.api.letsencrypt.org/directory
}

:80 {
    header -Server
}

cloud.skynewz.dev {
    # https://caddyserver.com/docs/caddyfile/directives/push
    push

    # https://caddyserver.com/docs/caddyfile/directives/encode
    encode zstd gzip

    # https://caddyserver.com/docs/caddyfile/directives/metrics
    metrics /metrics

    # https://caddyserver.com/docs/caddyfile/directives/tls
    tls {
        dns gandi {env.GANDI_API_TOKEN}
    }

    # https://caddyserver.com/docs/caddyfile/directives/header
    header {
        # Hide "Server: Caddy"
        -Server

        # prevent attacks such as Cross Site Scripting (XSS)
        Content-Security-Policy default-src 'self' cloud.skynewz.dev

        # enable the cross-site scripting (XSS) filter built into modern web browsers
        X-XSS-Protection 1; mode=block

        # ensures the connection cannot be establish through an insecure HTTP connection 
        Strict-Transport-Security max-age=31536000; includeSubDomains; preload

        # clickjacking protection
        X-Frame-Options DENY

        # provides clickjacking protection. Disable iframes
        X-Frame-Options: SAMEORIGIN

        # disable clients from sniffing the media type
        X-Content-Type-Options nosniff

        # keep referrer data off of HTTP connections
        Referrer-Policy no-referrer-when-downgrade
    }

    # https://caddyserver.com/docs/caddyfile/directives/respond
    # Replace backends health checks and provide one for this LB
    # respond /health 200

    # https://caddyserver.com/docs/caddyfile/directives/log
    log {
        output stdout
        format console
    }

    # https://caddyserver.com/docs/caddyfile/directives/reverse_proxy
    reverse_proxy * {
        # Specify backend here
        to 10.70.12.85:30438
        to 10.69.102.65:30438

        lb_policy round_robin
        lb_try_duration 1s
        lb_try_interval 250ms

        # health_path     /health # Backend health check path
        # health_port     80 # Default same as backend port
        # health_interval 10s
        # health_timeout  2s
        # health_status   200
        # health_body "OK"

        fail_duration 2s
        max_fails 2
        unhealthy_status 5xx
        unhealthy_latency 10s
        unhealthy_request_count 10
    }
}

https://crt.sh/?q=cloud.skynewz.dev&dir=^&sort=1&group=none
https://github.com/libdns/gandi/issues/1

[SkYNewZ]

Same with a very simple Caddyfile :

Oct 09 14:42:05 caddy systemd[1]: Started Caddy.
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1453805,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1503983,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.151538,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000338690"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1533282,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":4
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1533804,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.153854,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["cloud.skynewz.dev"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1541166,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.154134,"msg":"serving initial configuration"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1555028,"logger":"tls","msg":"cleaned up storage units"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1563451,"logger":"tls.obtain","msg":"acquiring lock","identifier":"cloud.skynewz.dev"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1569538,"logger":"tls.obtain","msg":"lock acquired","identifier":"cloud.skynewz.dev"}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.1688776,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 14:42:05 caddy caddy[5517]: {"level":"info","ts":1602254525.169309,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 14:42:06 caddy caddy[5517]: Oct 09 14:42:06 caddy caddy[5517]: {"level":"info","ts":1602254526.2846875,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Oct 09 14:44:08 caddy caddy[5517]: {"level":"error","ts":1602254648.374959,"logger":"tls.obtain","msg":"will retry","error":"[cloud.skynewz.dev] Obtain: [cloud.skynewz.dev] solving challenges: waiting for solver *certmagic.DNS01Solver to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/98841065/5602236743) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":123.217807825,"max_duration":2592000}
Oct 09 14:45:11 caddy caddy[5517]: {"level":"info","ts":1602254711.0881233,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}

cloud.skynewz.dev
tls [email protected] {
    dns gandi {env.GANDI_API_TOKEN}
}
respond "Hello, world!"

[mholt]

Hmmm, not sure, I don't use Gandi. Perhaps @obynio would be able to help debug?

[obynio]

@SkYNewZ I read through the logs you provided. The records TXT records _acme-challenge are created successfully so the Gandi plugin is working as expected. The rest depends of the ACME plugin within caddy which must fetch these DNS records to prove ownership of the domain.

I'm suspecting an issue related to DNS propagation. The ACME challenge must be cleared within 60 seconds but I believe the DNS propagation of the new TXT records haven't reached your DNS server by that time or the cache of your DNS server is not updated within 60s.

I would recommend to switch to another DNS server in your /etc/resolv.conf to confirm this hypothesis, I'm using 1.1.1.1 as a reference, give it a try.

[SkYNewZ]

Indeed, I'm using Scaleway and they set 127.0.0.53 as default name server. I think I shouldn't replace this line one /etc/resolv.conf. So I add new lines :

root@caddy:~# cat /etc/resolv.conf 
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 1.1.1.1
nameserver 1.0.0.1 
nameserver 127.0.0.53
options edns0

I can check that 1.1.1.1 is used :

root@caddy:~# dig cloud.skynewz.dev

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> cloud.skynewz.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35385
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;cloud.skynewz.dev.     IN  A

;; ANSWER SECTION:
cloud.skynewz.dev.  189 IN  A   51.15.139.15

;; Query time: 3 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Oct 09 20:59:03 UTC 2020
;; MSG SIZE  rcvd: 62

root@caddy:~# dig foo.cloud.skynewz.dev

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> foo.cloud.skynewz.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39890
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;foo.cloud.skynewz.dev.     IN  A

;; ANSWER SECTION:
foo.cloud.skynewz.dev.  300 IN  CNAME   cloud.skynewz.dev.
cloud.skynewz.dev.  300 IN  A   51.15.139.15

;; Query time: 88 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Oct 09 21:01:33 UTC 2020
;; MSG SIZE  rcvd: 80

But error persist and I can't understand why…

Oct 09 21:02:27 caddy systemd[1]: Started Caddy.
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7027378,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"warn","ts":1602277347.716023,"logger":"admin","msg":"admin endpoint disabled"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.717272,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f4af0"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.717332,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.717573,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv1","https_port":443}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7176151,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv1"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"warn","ts":1602277347.71767,"logger":"http","msg":"user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects","server_name":"srv0","interface":"tcp/:80"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7198467,"logger":"tls","msg":"cleaned up storage units"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"debug","ts":1602277347.7205827,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"debug","ts":1602277347.7208762,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.720912,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["cloud.skynewz.dev","*.cloud.skynewz.dev"]}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7237756,"logger":"tls.obtain","msg":"acquiring lock","identifier":"cloud.skynewz.dev"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7242815,"logger":"tls.obtain","msg":"lock acquired","identifier":"cloud.skynewz.dev"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7271712,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7287393,"msg":"serving initial configuration"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7298508,"logger":"tls.obtain","msg":"acquiring lock","identifier":"*.cloud.skynewz.dev"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.730628,"logger":"tls.obtain","msg":"lock acquired","identifier":"*.cloud.skynewz.dev"}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7374618,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7375004,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["cloud.skynewz.dev"]}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.7383351,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["*.cloud.skynewz.dev"]}
Oct 09 21:02:27 caddy caddy[5564]: {"level":"info","ts":1602277347.738657,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["*.cloud.skynewz.dev"]}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"debug","ts":1602277348.316968,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 21:02:28 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"debug","ts":1602277348.4472065,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 09 Oct 2020 21:02:28 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103tKZ9BYHX9CHKuDYaLO7hyBJvJG3tIzSHdBuhd1gg__k"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"debug","ts":1602277348.7309587,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["98867609"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["347"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 21:02:28 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/98867609/5606685176"],"Replay-Nonce":["0104RILb1icACLYslvFLSfC0Dx9wWQN3-OnMqGmJ4oUtOhU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"debug","ts":1602277348.8930273,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/7781251552","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["98867609"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["795"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 21:02:28 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0102DbFBTwGZqY0cZjOwzWh5ZhYSe-AjMDdo9OuL98hNWwI"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"debug","ts":1602277348.8936837,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"http-01"}
Oct 09 21:02:28 caddy caddy[5564]: {"level":"info","ts":1602277348.8937902,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Oct 09 21:02:29 caddy caddy[5564]: {"level":"debug","ts":1602277349.1467714,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Fri, 09 Oct 2020 21:02:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002-bJ-e1v11Yh8icaiOtu2fF7jnNYsx_Dmr8IM3wbEuTk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:29 caddy caddy[5564]: {"level":"debug","ts":1602277349.368015,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["98867609"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["339"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 21:02:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/98867609/5606685270"],"Replay-Nonce":["0001c9KngWhl_ieEZ707FQwXQC_knMxAccPvGKklwXQ7sao"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:29 caddy caddy[5564]: {"level":"debug","ts":1602277349.578204,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/7781087010","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.2.0 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["98867609"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["389"],"Content-Type":["application/json"],"Date":["Fri, 09 Oct 2020 21:02:29 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0002J4LWQOOAoGx28ir9Ii0PcMor2xqa6K1aKfUuUGS_bCw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
Oct 09 21:02:29 caddy caddy[5564]: {"level":"info","ts":1602277349.5784795,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"*.cloud.skynewz.dev","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}

[mholt]

@SkYNewZ I might be missing something, but I don't see any error in your logs now?

[SkYNewZ]

@SkYNewZ I might be missing something, but I don't see any error in your logs now?

Indeed, I truncate too early 🤦🏻‍♂️
But is always looping, and spamming my DNS zone of acme*
I'm almost sure that is not Caddy's fault now but I don't know how to set up correctly on this instance regarding this DNS issue…

[mholt]

Okie. Wellllll at this point I think it's pretty clear it's not a bug in Caddy, so I'll close this. Feel free to continue discussion if needed!

[SkYNewZ]

Yes sorry for the inconvenience… Do you have an idea about can I fix it ?

[mholt]

I am not sure... there are many factors on your network, system, infrastructure, etc, that could be affecting it. Good luck! You can try asking on our forums: https://caddy.community

[SkYNewZ]

Thanks. I have noticed the community with I also facing this issue on my laptop. Weird

[SkYNewZ]

Nor with another root domain… Do you think its a Gandi issue ?

[SkYNewZ]

@mholt UPDATE !

I don't have this issue using Caddy v.2.1.0 and Go 1.14 !

root@055e6da050ce:/code# ./caddy version
v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=
root@055e6da050ce:/code# ./caddy run
2020/10/10 15:38:55.726 INFO    using adjacent Caddyfile
2020/10/10 15:38:55.731 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/10/10 15:38:55 [INFO][cache:0xc00013bd40] Started certificate maintenance routine
2020/10/10 15:38:55.731 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/10/10 15:38:55.743 INFO    http    enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2020/10/10 15:38:55.745 INFO    http    enabling automatic TLS certificate management   {"domains": ["*.cloud.skynewz.dev", "cloud.skynewz.dev"]}
2020/10/10 15:38:55.748 INFO    tls cleaned up storage units
2020/10/10 15:38:55.748 INFO    autosaved config    {"file": "/root/.config/caddy/autosave.json"}
2020/10/10 15:38:55.748 INFO    serving initial configuration
2020/10/10 15:38:55 [INFO][cloud.skynewz.dev] Obtain certificate; acquiring lock...
2020/10/10 15:38:55 [INFO][*.cloud.skynewz.dev] Obtain certificate; acquiring lock...
2020/10/10 15:38:55 [INFO][cloud.skynewz.dev] Obtain: Lock acquired; proceeding...
2020/10/10 15:38:55 [INFO][*.cloud.skynewz.dev] Obtain: Lock acquired; proceeding...
2020/10/10 15:38:56 [INFO] acme: Registering account for [email protected]
2020/10/10 15:38:56 [INFO][cloud.skynewz.dev] Waiting on rate limiter...
2020/10/10 15:38:56 [INFO][cloud.skynewz.dev] Done waiting
2020/10/10 15:38:56 [INFO] [cloud.skynewz.dev] acme: Obtaining bundled SAN certificate given a CSR
2020/10/10 15:38:56 [INFO][*.cloud.skynewz.dev] Waiting on rate limiter...
2020/10/10 15:38:56 [INFO][*.cloud.skynewz.dev] Done waiting
2020/10/10 15:38:56 [INFO] [*.cloud.skynewz.dev] acme: Obtaining bundled SAN certificate given a CSR
2020/10/10 15:38:57 [INFO] [cloud.skynewz.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/7797770761
2020/10/10 15:38:57 [INFO] [cloud.skynewz.dev] acme: Could not find solver for: tls-alpn-01
2020/10/10 15:38:57 [INFO] [cloud.skynewz.dev] acme: Could not find solver for: http-01
2020/10/10 15:38:57 [INFO] [cloud.skynewz.dev] acme: use dns-01 solver
2020/10/10 15:38:57 [INFO] [cloud.skynewz.dev] acme: Preparing to solve DNS-01
2020/10/10 15:38:57 [INFO] [*.cloud.skynewz.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/7797770792
2020/10/10 15:38:57 [INFO] [*.cloud.skynewz.dev] acme: use dns-01 solver
2020/10/10 15:38:57 [INFO] [*.cloud.skynewz.dev] acme: Preparing to solve DNS-01
2020/10/10 15:38:58 [INFO] [cloud.skynewz.dev] acme: Trying to solve DNS-01
2020/10/10 15:38:58 [INFO] [cloud.skynewz.dev] acme: Checking DNS record propagation using [192.168.65.1:53]
2020/10/10 15:38:58 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2020/10/10 15:38:58 [INFO] [cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:38:58 [INFO] [*.cloud.skynewz.dev] acme: Trying to solve DNS-01
2020/10/10 15:38:58 [INFO] [*.cloud.skynewz.dev] acme: Checking DNS record propagation using [192.168.65.1:53]
2020/10/10 15:38:58 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2020/10/10 15:38:58 [INFO] [*.cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:00 [INFO] [cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:00 [INFO] [*.cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:03 [INFO] [*.cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:05 [INFO] [*.cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:06 [INFO] [cloud.skynewz.dev] The server validated our request
2020/10/10 15:39:06 [INFO] [cloud.skynewz.dev] acme: Cleaning DNS-01 challenge
2020/10/10 15:39:07 [INFO] [*.cloud.skynewz.dev] acme: Waiting for DNS record propagation.
2020/10/10 15:39:07 [INFO] [cloud.skynewz.dev] acme: Validations succeeded; requesting certificates
2020/10/10 15:39:07 [INFO] [cloud.skynewz.dev] Server responded with a certificate.
2020/10/10 15:39:07 [INFO][cloud.skynewz.dev] Certificate obtained successfully
2020/10/10 15:39:07 [INFO][cloud.skynewz.dev] Obtain: Releasing lock
2020/10/10 15:39:12 [INFO] [*.cloud.skynewz.dev] The server validated our request
2020/10/10 15:39:12 [INFO] [*.cloud.skynewz.dev] acme: Cleaning DNS-01 challenge
2020/10/10 15:39:13 [INFO] [*.cloud.skynewz.dev] acme: Validations succeeded; requesting certificates
2020/10/10 15:39:14 [INFO] [*.cloud.skynewz.dev] Server responded with a certificate.
2020/10/10 15:39:14 [INFO][*.cloud.skynewz.dev] Certificate obtained successfully
2020/10/10 15:39:14 [INFO][*.cloud.skynewz.dev] Obtain: Releasing lock

Is is a acmez issue ?

[mholt]

That's quite interesting! Possibly, but we borrowed the same DNS utility code from lego. I'm out of the office this weekend, would you be able to investigate a little more? That would help speed up a fix!

[SkYNewZ]

I think the issue come from the DNS zone create acmez. Here you can see the record its different from acmez and lego on my skynewz.dev zone:

- _acme-challenge.cloud.skynewz.dev 10800 IN TXT "tTmFZ_um62V4jNCnirwWB533pq-esRsOCxfDmWvi1As"
+ _acme-challenge.cloud 10800 IN TXT "4EUlmNLWvtxXSmCqiYpj11LxdWVLzpslwmZLCQGDcb0"
_acme-challenge.cloud 10800 IN TXT "LSYS4kAXLHRLqCq71wIGU9vwkv3A_oyAgwMDGtyOiTM"

Caddy (with acmez) create a record called _acme-challenge.<WANTED_RECORD>.<ROOT_DOMAIN> whereas Caddy (with lego) create a record called _acme-challenge.<WANTED_RECORD>. This is probably why Caddy itself cannot found it ?

[SkYNewZ]

Works on my laptop :

~/Sources/caddyv2/test using ☁️  default/iwc-apipy-ss-int-dtep 
➜ sudo vim /private/etc/hosts                 
Password:
Sorry, try again.
Password:

~/Sources/caddyv2/test using ☁️  default/iwc-apipy-ss-int-dtep 
➜ cat /private/etc/hosts 
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost
127.0.0.1 docker.local

# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section
#
# Remove this !
127.0.0.1 cloud.skynewz.dev


~/Sources/caddyv2/test using ☁️  default/iwc-apipy-ss-int-dtep 
➜ ping cloud.skynewz.dev            
PING cloud.skynewz.dev (127.0.0.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
^C
--- cloud.skynewz.dev ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

~/Sources/caddyv2/test using ☁️  default/iwc-apipy-ss-int-dtep 
➜ https https://cloud.skynewz.dev 
HTTP/1.1 200 OK
Content-Length: 13
Date: Sat, 10 Oct 2020 15:52:21 GMT
Server: Caddy

Hello, world!

[SkYNewZ]

That's quite interesting! Possibly, but we borrowed the same DNS utility code from lego. I'm out of the office this weekend, would you be able to investigate a little more? That would help speed up a fix!

I don't specially need latest version and yes I think a will use v2.1.0 with Go1.14. Good luck for the fix.

Maybe acmez itself creates the wrong TXT record

[obynio]

@SkYNewZ the record itself is created by the Gandi plugin, but is follows the DNS-01 challenge RFC so I doubt _acme-challenge.cloud comply with it. I think it may be an issue with Go 1.15 and the ACME code but it's just a gut feeling.

[SkYNewZ]

After my test the record _acme-challenge.cloud works but not the _acme-challenge.cloud.skynewz.dev, even if i agree its not follows the RFC...
I don't know any more unfortunately. It's not very important for me, I'm now using the lower version, but I'm missing the push and metrics directives which doesn't seem to exist in version 2.2.0...

[SkYNewZ]

Using the v2.1.0 I have this bug...
@mholt Do you plan/have time to do something for this issue ? Without wishing to be rude for sure

[mholt]

@SkYNewZ I don't think an issue with record creation belongs in this repo, can you please file an issue with CertMagic or the gandi plugin repository, wherever it is that the bug exists?

[obynio]

Issue solved https://github.com/caddyserver/certmagic/issues/105#issuecomment-708614567