Caddy: Custom DNS resolver for ACME

[Gorian]

1. What would you like to have changed?

Add an option to use an external/custom DNS resolve for verifying acme challenges. Such as Traefik has: https://docs.traefik.io/configuration/acme/#resolvers

2. Why is this feature a useful, necessary, and/or important addition to this project?

As of now, caddy is unusable with acme and split DNS.

Scenario:
I own domain.com. domain.com is hosted with cloudflare. Following best practices, my internal domain is a dedicated sub-domain of my IANNA registered domain, in.domain.com. I am running Active Directory, so I have 2 windows DNS servers acting as authoritative DNS servers for my zone, in.domain.com. However, due to some software limitations, such as ceph's s3 api gateway only binding to a single hostname, and external sites using s3 by uploading a file and then merely linking to it, or running a music server and using the same DNS name for said music server so it works when I'm at home, and when I'm away from home, i run split DNS. This works fine 99% of the time for me. Rather than query Cloudflare for music.domain.com or s3.domain.com and getting my external IP returned, my windows DNS server return an authoritative response with internal IPs instead.

Where this breaks though, is caddy. I am taking advantage of caddy's dynamic configuration by using SRV records. For example, caddy load balances by multiple s3 gateway nodes with srv://_http._tcp.s3.in.domain.com, and this works. But in order to do that, it needs to resolve that DNS name with my internal DNS servers. This is not a problem, since DNS is correctly configured in /etc/resolv.conf. However, the problem comes in trying to use DNS-01 with LetsEncrypt/ACME. The lego client locally verifies that the TXT record was made using your preferred API, before telling LetsEncrypt to verify it. It uses these resolvers, in order, depending on whether they are available or not:

• Custom resolvers (set by the application) I.E. --dns.resolvers value
• /etc/resolve.conf (if length of file does not equal 0)
• Google's DNS servers

because I require my internal DNS to resolve my SRV records, I can't just just set my DNS to an external resolver.

The end result is, caddy waits infinitely for verification _acme-challenge.domain.com was made, but internal DNS returns an authoritative response, not CloudFlare.

3. What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

Right now, I have a seperate, dedicated server that uses certbot + a custom deploy hook written in bash to get certs with certbot + cloudflare plugin, where certbot uses it's own resolver, and not my internal one, to verify the challenge, and my custom deploy hook them concatenates the public and private keys into a shared file directory that is loaded by caddy using the load TLS directive. This works, but I'm not actually using the built-in TLS support, meaning I have to manually override things, such as binding to port 80 vs. 2015 that caddy likes to bind to if you aren't using LetsEncrypt within the application.

4. Please link to any relevant issues, pull requests, or other discussions.

• lego nameserver.go
• lego github (with relevant usage)
• specific lego issue

[francislavoie]

I guess this could be done via an option on the tls directive? Since it looks like lego supports it, should be a simple config passthrough, unless I'm mistaken. PRs welcome if you want to take a crack at it!

[Gorian]

I've never got around to writing any Go yet :/ Maybe at some point I might take a crack at it, when I have time to sit down and learn some Go

[the-maldridge]

I looked at seeing if this could just be added directly to lego (#828), but there is pushback against putting it in lego directly. imho because it deals with the configuration of the challenge it should be pulled from the environment, rather than being another token in the tls block.

@idez Had the wrong tab open and added this to another issue by mistake.

[altano]

I tried setting up Traefik around the v2 release and figuring out how to use it is now a crazy mess. I kept hearing that it is so much easier to use as a Docker reverse proxy, but between Caddy's simple config and the caddy-docker-proxy package it was extremely easy to set up Caddy.

EXCEPT this lack of split-dns support. And since I can't seem to find a workaround for this issue, I guess it's back to Traefik for me.

I look forward to this getting implemented in Caddy 2 if you all are able to get to it. Thanks!

[mholt]

We can probably work this in, but right now our development priorities are constrained to what companies are willing to pay for -- or the community can propose, discuss, and contribute this feature with a PR. I'm currently out of runway so I cannot work on it at this time :( -- this feature either needs to be sponsored or a company needs to hire for it.

[altano]

Of course. Thank you for the time you've put into Caddy and for being open to someone possibly sponsoring this work in the future. Best of luck!

[frimik]

Also, if your server CANNOT call out to authoritative DNS servers (firewall) - you cannot use the acme support in caddy because it will constantly time out (udp) requests against a firewalled external dns server.

Same issue as cert-manager used to have... https://github.com/jetstack/cert-manager/commit/3eaca6a318c47e7433ceaa94f1a5aa14d4442681

[mholt]

@frimik It's not hard... if someone wants to contribute it, by all means, submit a PR, or we'll have to find funding for me to add this feature right now if you want me to do it.

[frimik]

I understand @mholt . Not sure how my needs turn out yet.

Perhaps this little iptables hack can help someone else who's behind an egress-blocking firewall :hammer: .

This will make the requests trying to reach any DNS server outside of your LAN hit a specific internal DNS IP instead and will make Caddy complete its DNS-01 process :).

#!/bin/bash

DNS="10.46.10.133"
LAN_CIDR="10.0.0.0/8"

iptables -A OUTPUT -d 127.0.0.0/8,${LAN_CIDR} -p udp -m udp --dport 53 -j ACCEPT
iptables -A OUTPUT ! -d 127.0.0.0/32 -p udp -m udp --dport 53 -j DNAT --to-destination ${DNS}
iptables -A OUTPUT ! -d ${LAN_CIDR} -p udp -m udp --dport 53 -j DNAT --to-destination ${DNS}
iptables -A OUTPUT ! -d 127.0.0.0/8 -p tcp -m tcp --dport 53 -j DNAT --to-destination ${DNS}
iptables -A OUTPUT ! -d ${LAN_CIDR} -p tcp -m tcp --dport 53 -j DNAT --to-destination ${DNS}

[mholt]

@frimik Thanks for sharing that workaround in the meantime!

[mholt]

This is now implemented, you can use the JSON config to set resolvers in the DNS challenge config. They will be used with preference over the system resolvers. Edit: You can now use Caddyfile too: https://github.com/caddyserver/caddy/commit/24f34780b61fc82df6507cf95535c3dfc9f78058

[Gorian]

@mholt been a while since I've been involved with anything to do with caddy or had time to work on my own homelab stuff, but I saw that this was updated and wanted to say thanks for implementing it! Definitely a reason to keep using this product :)