Caddy: Add option to force issuance of a wildcard cert

[smtalk]

It be great to have an option to force issuance of a wildcard cert for non-wildcard vhosts. Something like:

"automation": { "policies": [{ "wildcard": true }]}

[mholt]

To clarify, OP is asking for a feature that, when Caddy is asked to manage a certificate for foo.example.com, it actually manages a certificate for *.example.com.

[smtalk]

@mholt, I'd expect it to do the same for example.com, so that it'd actually manage a cert for *.example.com. Thanks!

[mholt]

Hmm okay, got it.

Another user needs it the other way: https://caddy.community/t/host-matchers-and-tls-certificates/7340?u=matt

This will be tricky to figure out...

[smtalk]

Maybe just something like:
"automation": { "policies": [{ "name_to_use": "*.example.com" }]}

Then it'd be a user-decision what to use there :) We'd just auto-generate the config for any domain needing wildcard. If there is a cert loaded already - Caddy would just skip generation of it.

So, if *.b.domain.com is already there there, a.b.domain.com vhost having *.b.domain.com auto-defined to configs, would just use Caddy's already loaded cert, instead of creation of a new one.

[shinenelson]

If there is a cert loaded already - Caddy would just skip generation of it.

So, if *.b.domain.com is already there there, a.b.domain.com vhost having *.b.domain.com auto-defined to configs, would just use Caddy's already loaded cert, instead of creation of a new one.

Unfortunately, though I had the exact same scenario, it didn't work for me. I was thinking that this was already built-in though.

[mholt]

Currently, Caddy manages certificates for the literal names you tell it to manage (implicitly or explicitly). I really like the simplicity/straightforwardness of that, so I'm hoping there's a way to do what you want without adding transformations that happen under the hood to the domain name.

[shinenelson]

Maybe just something like:
"automation": { "policies": [{ "name_to_use": "*.example.com" }]}

That brings about to the definition of SANs I believe? :thinking:

[shinenelson]

I really like the simplicity/straightforwardness of that

I love it too. Its simple, yet powerful ( especially with the API ). That's what I like about caddy.

If it has to go through transformations and stuff like that that would take that simplicity away from caddy, I'd take my request back until there's more pressure from a larger audience for the feature.

[whalehub]

@mholt I'm very confused as to why this feature is being described as "tricky to figure out" - isn't this the purpose of the wildcard option in Caddy v1's tls directive? In my experience of using Caddy for a couple years, if that option is present in the site config of a subdomain, Caddy will generate a wildcard certificate for its parent domain (e.g. with the SAN *.example.com) and reuse that certificate for all other site configs which use the wildcard option and share the parent domain example.com. If the option isn't present in a site config, Caddy will generate a site-specific certificate (e.g. with the SAN subdomain.example.com) instead.

[mholt]

@whalehub

I'm very confused as to why this feature is being described as "tricky to figure out" - isn't this the purpose of the wildcard option in Caddy v1's tls directive?

Have you tried to implement this feature (in a correct way that doesn't break anything and works as expected for everyone)? I'm not surprised if it looks easy on the surface, as is the case with all features. :)

In my experience of using Caddy for a couple years,

You've been using Caddy 1, but we are talking about Caddy 2, which is totally different.

Caddy will generate a wildcard certificate for its parent domain (e.g. with the SAN *.example.com) and reuse that certificate for all other site configs which use the wildcard option and share the parent domain example.com.

Yes, but as you can see from reading above, we have one user who wants it to work this way (sub.example.com -> *.example.com) , while we have another user who wants it to work a different way (example.com -> *.example.com).

You can start appreciating the problem by reading about how TLS automation policies work, and I would be happy to entertain ideas for solutions until I can get around to implementing it!

[whalehub]

@mholt I'm sorry, but I don't see how this is a difficult problem to solve. You could satisfy both users by making a minor adjustment to the way things work in Caddy v1 by including the base domain in the LetsEncrypt certificate request whenever the hypothetical wildcard option in Caddy v2 is specified in a site's config.

There are essentially four hypothetical scenarios we can look at.

1. A site config is for a base domain (example.org) and the wildcard directive is absent from it. Caddy proceeds to request a certificate with the SAN example.org from LetsEncrypt.

2. A site config is for a subdomain (sub.example.org) and the wildcard directive is absent from it. Caddy proceeds to request a certificate with the SAN sub.example.org from LetsEncrypt.

3. A site config is for a base domain (example.org) and the wildcard directive is present. Caddy proceeds to request a certificate with the SANs example.org and *.example.org LetsEncrypt.

4. A site config is for a subdomain (sub.example.org) and the wildcard directive is present. Caddy proceeds to request a certificate with the SANs example.org and *.example.org from LetsEncrypt.

The ACME client that Caddy uses internally - lego - has been able to do this for years. What am I not seeing here?

[mholt]

@whalehub Please submit a pull request :) If it's that simple, we can get it merged in sooner rather than later.

There _are_ complexities to this problem, but it's also a matter of timeline and priorities, so it's rude or at least impolite to burst into an issue and say "This doesn't seem so hard ... [other projects] can do this for years now, why isn't this done yet?" ... just so you know.

[whalehub]

@mholt I don't appreciate you putting words in my mouth; I think that's a pretty rude thing to do. What I said is that the ACME client which Caddy uses internally has been able to request wildcard certificates from LetsEncrypt with the base domain as an additional SAN for years. It follows logically that Caddy should be able to do it as well.

I think it would be a more productive use of your time to explain the complexities that you see with exposing an existing feature of Caddy's internals in Caddy's configuration files so that a solution to them can be brainstormed rather than passive-aggressively asking me to code this functionality for you.

[francislavoie]

@whalehub Matt has absolutely no obligations to implement any feature. He does it because he's passionate about moving this project forwards.

If it's such an important feature for you, why don't you consider sponsoring him so he can afford to raise the priority of this issue?

[whalehub]

@francislavoie I don't think that anyone is obligated to implement this or any other feature for me, but I do think that it's reasonable to expect that future versions of a software at a bare minimum maintain the feature set of previous versions unless there is a good reason to remove some of them (e.g. the deprecation of older TLS versions for security reasons in Caddy v0.11.5).

In software development circles, that idea is known as "backwards compatibility" and even @mholt seems to see the value in it since he published this article on the wiki for Caddy v1. As a long-time user of Caddy v1, it would be very disappointing if Caddy v2 ended up being worse than Caddy v1 rather than much better than it.

[francislavoie]

You being confrontational is completely unproductive. Please reassess the tone you're taking in these threads, it's really not appreciated.

To clarify, the article you linked is a compatibility guarantee for _v1_, as in v1.x.x, not a guarantee for anything past that point.

v2 is a complete rewrite of Caddy from the ground up, with completely different architecture based on the 5 years of experience gained from building Caddy up to this point. Tradeoffs were made, and Caddy v2 is much more powerful in almost every way than v1.

[whalehub]

@francislavoie I don't think I've been confrontational in my posts. I voiced an issue that I see with the discussion around this feature in good faith, expanded on why I think it's an issue when my original post was challenged and have only received passive-aggressive, condescending and accusatory responses in return since then.

I should note that I personally contributed to Caddy's ecosystem in the past by spending several hours of my time going through every single plugin repository on GitHub and opening almost 100 pull requests in order to bring Caddy's entire plugin ecosystem up-to-date with Caddy's repository changes in 2019: https://github.com/epicagency/caddy-expires/issues/6

In light of that, you can imagine that I don't appreciate being treated as a bad faith actor when that couldn't be further from the truth.

[mholt]

@whalehub

I often feel the weight of many user's demands from working on this project. Over the last 5 years, I've interacted with thousands of users and customers, each who have different requirements. Caddy (and especially Caddy 2) exposes thousands of parameters and hundreds of control surfaces, all which get deployed into countless environments; the result is millions of possible combinations. Multiply that with intricacies of business expectations, funding concerns (which directs prioritization of issues), and user feedback (OSS projects have no "shields" or filters between the users and developers like traditional corporate software, so developers are very directly exposed to every iota of feedback, like being out in freezing rain with no umbrella)... the incessant tugs in every direction to make the software perfect for yet another user does indeed sometimes amount to an almost unbearable load.

That kind of pressure makes me more prone to mistakes, which I've made my fair share of with this project over the last 5 years. (And unfortunately, those seem to be the only thing about the project that people remember and focus on, even long after the mistakes have been remedied. It's impossible to repent in this industry.)

When I first read your comments, they came across as belittling and insensitive more than helpful. I felt added pressure. I apparently read too much into your posts and not enough into your intentions.

The way I responded to you, and to several others in a similar boat lately, is one of my mistakes, but one which I tend to keep repeating. I'm sorry.

I was recently reminded in a very good story which I love that I would do well to identify my allies before identifying enemies.

Let me come at this again, with the mindset that you're my ally in this project, not my adversary: it hasn't been implemented yet because it hasn't been implemented yet.

It's that simple. I don't have a better reason to explain, and although there may be one, I'm sure neither you nor anyone else would want to be bothered with it.

Now, to answer your original question:

isn't this the purpose of the wildcard option in Caddy v1's tls directive?

directly and succinctly, without masquerading it or reading into anything else: Yes, it is.

[whalehub]

@mholt Thank you for giving me some insight into what it's like to develop a project of this size. I apologize if I came off as belittling, that was not my intention. As someone who is not a software developer themselves but "only" a Linux Sysadmin, I was coming from a place of genuine confusion when I found out that the wildcard option I frequently use in Caddy v1 does not yet exist in Caddy v2.

[shinenelson]

Wow, guys, this escalated pretty quickly.

@whalehub, if I may ( fellow sysadmin here too ), let me break this down for you the way I understood it and add to what Matt explained already.

---

Disclaimer : I haven't used caddy v1 at all, so, my understanding of that part might be lacking. However, I've used caddy v2 extensively ( even though for testing ) in different scenarios trying to integrate it for a project that I was building up.

---

I was coming from a place of genuine confusion when I found out that the wildcard option I frequently use in Caddy v1 does not yet exist in Caddy v2

False. The wildcard option does exist in caddy v2 ( including the tls directive in the Caddyfile ). Matt linked to the TLS Automation Policies documentation too.

I see you mention SANs a lot here. I understand the feature when it comes to TLS certificates. However, Matt mentioned on a community post that caddy recommends _against_ managing multi-SAN certificates and I agree with pretty much everything in the linked document as well. More justifications are on the community post ( I'm not sure if that is what you meant in your posts, but that's what I understood from it ). That's also why I too brought it up in a previous comment and gave up.

---

This issue was not about generating wildcard certificates at all. It was about an algorithm / logic to decide when to generate a wildcard certificate and when not to.

smtalk ( who opened this issue ) wanted a wildcard TLS certificate ( *.example.com ) to be generated even if the host matcher was a subdomain ( or even an apex domain ). That is, if foo.example.com ( or example.com ) is in the host matcher.

However, I wanted a host matcher for foo.example.com to match to the wildcard TLS certificate ( *.example.com ) that I had already automated to be generated without generating a new certificate for foo.example.com.

And then I also came up with another use-case ( even though I didn't need it ) where someone else might actually want the opposite happening too.

---

I hope you can now see the conflicting use-cases / contexts which make the implementation of this feature ( without breaking a lot of web-servers that are already deployed to production ) complicated.

Let me re-iterate my agreement with Matt from a previous comment :

Currently, Caddy manages certificates for the literal names you tell it to manage (implicitly or explicitly). I really like the simplicity/straightforwardness of that, so I'm hoping there's a way to do what you want without adding transformations that happen under the hood to the domain name.

I love it too. Its simple, yet powerful ( especially with the API ). That's what I like about caddy.

If it has to go through transformations and stuff like that that would take that simplicity away from caddy, I'd take my request back until there's more pressure from a larger audience for the feature.

[whalehub]

False. The wildcard option does exist in caddy v2 ( including the tls directive in the Caddyfile ). Matt linked to the TLS Automation Policies documentation too.

@shinenelson I don't know whether or not they have been updated since you last visited them, but I cannot find any mention of a wildcard option on the documentation page for the tls directive in Caddy v2's Caddyfile or on the page which you linked in your response. I can't seem to find any mention of a dns option on these pages either, which is required to issue wildcard certificates in Caddy v1.

[shinenelson]

Oh, wait, my bad. I think I was half-sleeping or something when I typed that out.

You are right, the wildcard option is probably not supported in the Caddyfile ( I'm not sure about this because I haven't used it myself ), but it is very well supported by Caddy though. It is available in the native JSON configuration, for sure.

I'm sorry for quoting the wrong documentation. I assumed the native JSON configuration in my head even though I mentioned Caddyfile in the comment. Also, I think the context of this issue was also around the native JSON configuration since there were suggestions in JSON with possible implementations.

[whitestrake]

Just weighing in here to say that I think *.example.com and example.com should not be conflated, nor should they be included on the same certificate. Caddy can manage both separately, _easily_, if required. v1 does not conflate the two either, unless I'm mistaken.

Apart from not being incredibly keen on the idea of multi-SAN certificates (see https://docs.https.dev/acme-ops#use-one-name-per-certificate), they're completely different levels of the domain name.

If we say example.com should fetch *.example.com, why wouldn't sub.example.com fetch *.sub.example.com? Where and how does the line get drawn on this logic? What about example.com.au with this feature - do we need to start maintaining lists of TLDs and 2LDs to ensure we do get a wildcard subdomain when it's appropriate and don't when it's not?

Honestly, I think on a lot of levels it's simpler and better if, when you want to use sub.example.com and want a *.example.com cert for it, you could specify that, just like v1. If your site uses example.com and *.example.com, all the more power - Caddy will happily manage them anyway, separately.

@whalehub

I can't seem to find any mention of a dns option on these pages either, which is required to issue wildcard certificates in Caddy v1.

DNS configuration is specified per issuer, for ACME see https://caddyserver.com/docs/modules/tls.issuance.acme (look in the challenges key).

[francislavoie]

Coming back to this later, I don't think there's really anything to be done here.

If you want sub.example.com to have a wildcard cert, then write a Caddyfile like this:

*.example.com, example.com {
    @sub {
        host sub.example.com
    }
    handle @sub {
        # directives for sub.example.com...
    }

    # directives for any other host...
}

Caddy will fetch a wildcard cert for *.example.com and a regular cert for example.com and handle both in the same server block. You can use the host matcher to implement per-host handling if you need to, without it fetching a cert for that specific host.

[lflare]

New to this discussion, but I would really like the Caddy v1's ability to specify the following,

tls {
    dns cloudflare
    wildcard
}

However, I would also like to add that @francislavoie's little Caddyfile example seems to work wonders for me, for now I will stick with it, though it isn't really the most elegant of options I feel.

Coming back to this later, I don't think there's really anything to be done here.

If you want sub.example.com to have a wildcard cert, then write a Caddyfile like this:

*.example.com, example.com {
  @sub {
      host sub.example.com
  }
  handle @sub {
      # directives for sub.example.com...
  }

  # directives for any other host...
}

Caddy will fetch a wildcard cert for *.example.com and a regular cert for example.com and handle both in the same server block. You can use the host matcher to implement per-host handling if you need to, without it fetching a cert for that specific host.

[francislavoie]

FWIW Caddy v2.1 will make it a bit less ugly by allowing single-line matcher syntax:

*.example.com, example.com {
    @sub host sub.example.com
    handle @sub {
        # directives for sub.example.com...
    }

    # directives for any other host...
}

I still don't think the added complexity of a wildcard option would really add much value here. Just use an explicit site label.

[mholt]

One problem with the way v1 did it is it's not obvious what it is doing or what it means. I think the v2 way is much better.

[SvenDowideit]

@mholt I wonder if we can re-open this discussion.

in caddy-docker-proxy, this seems to cause every container to need to have the caddy: *.example.com, example.com label - which is really not desirable.

wrt "the old syntax wasn't intention revealing" I really would love the certificate policy to be separable from the routing policy - as my devs and I have many short lived containers with weird names, and having those use the wildcard certs is significantly preferable.

for eg, to be able to say

{
  tls {
      *.loc.alho.st {
         dns gandidns {env.GANDITOKEN}
         wildcard true
     }
      *.myother.com {
         dns route53 {env.ROUTE53TOKEN}
         wildcard true
     }
}

happy_einstein.loc.alho.st {
reverse_proxy 192.168.2.2:8080
}

reverent_pike.myother.com {
    ...
}

and for any other container that later reverse proxies to any domain that is matched by the tls policy to use those settings

looking at the json, its hard for me to see that its doing the right thing too

(dns_api_gandi) {
        tls {
                dns gandi {env.GANDIV5_API_KEY}
        }
}
*.loc.alho.st loc.alho.st {
        import dns_api_gandi
        @testing_loc_alho_st {
                host testing.loc.alho.st
        }
        route @testing_loc_alho_st {
                respond Testing
        }
}
sven.loc.alho.st {
        respond SVEN
}

results in

tls.automation....
            "subjects": ["*.loc.alho.st", "loc.alho.st"],

and no tls settings for sven.loc.alho.st BUT,

2020/08/25 07:18:23 [INFO] [sven.loc.alho.st] acme: Cleaning DNS-01 challenge

ie, even though there is nothing in the caddyfile, or in the json to say, use dns based tls for the cert, it has done so (but it hasn't use the wildcard setting, just the tls dns TOKEN part.)

mmm, that makes me think there are 2 issues - 1. the tls policy isn't in fact doing what it says, and 2. it would be excellent (and plausible) to separate the tls policy in the caddyfile, in such a way that this renders down to json that then makes sense.

IDK - yes, its complicated, but I think it could e easier.

[mholt]

@SvenDowideit

I really would love the certificate policy to be separable from the routing policy

Just so you know, that's exactly how the JSON config works. Its cert automation config is separate from its TLS connection and cert selection config, which is all separate from the HTTP routing config. The JSON really is a good structure for high levels of customization.

The point of the Caddyfile is that logic pertaining to a particular domain name or site address is contained within the block for that site address.

If you need the flexibility of the JSON, just use JSON.

Anyway, it's not clear to me from your post whether this is a separate issue (like a bug report) or whether it's hypothetical. It reads to me as if both kinds of things got combined there and I'm a bit confused. Can you please open a new issue to clearly explain the bug if there is one?