Caddy: v2: Caddyfile enhancements

[mholt]

The Caddyfile is one of the last things to be worked on in v2 since it is just a thin layer over Caddy 2's config, in fact, it is a separate plugin (whereas in v1, it was the central mode of configuration). What we have at this point was just brought over from v1 with very few changes -- most of the changes are totally internal (like the fact that it just generates JSON now). The matchers are the only real external change. A good one, but clunky.

Anyway, it's time to get it up to speed, and get it documented, so people will actually want to use Caddy again.

There are a number of deficiencies in the Caddyfile from v1 and in v2 currently that have been studied over the last few months (and years, frankly), which we now have a chance to improve on. If I have time I will try to link to them here, but if you've done anything mildly advanced with the Caddyfile, you know what I'm talking about.

I don't want to sacrifice the Caddyfile's simplicity and usability, though, either.

Since we're still in beta, now is the time to make breaking changes to get it right. Also, since the current Caddyfile documentation is scant, I suspect very few people will be impacted by the changes.

These solutions/enhancements could go a lot of different ways, but here's what I've settled on as of now:

1) Path matching will be an exact match instead of a prefix match. Prefix matching can still be done simply by ending in *.

Example: Proxying all API endpoints, before:

reverse_proxy /api/ localhost:8080

And after:

reverse_proxy /api/* localhost:8080

2) Within multiple instances of a directive, the Caddyfile adapter will sort them in descending order of specificity of their path matcher. (Specificity is hard/impossible to define with other matchers.)

Before, this example would be impossible, since the first line would always match those in the second line, and would never evaluate the second line since proxying is terminal:

reverse_proxy /api/ localhost:8081
reverse_proxy /api/v1/ localhost:8080

After this change, the above will be valid, since it will be sorted in the other order: first matching handler wins.

3) The rewrite directive will be mutually exclusive -- only the first matching rewrite will be executed. It will not "rehandle" -- this means rewrites do not cascade.

Example: Our new v2 documentation site uses something like this:

rewrite /docs/json/* /docs/json/index.html
rewrite /docs/*      /docs/index.html

But this ends up redundant because the first rewrite makes it into the second rewrite, since rewrites "rehandle". With the change, only the first rewrite would be evaluated for requests to /docs/json/*.

4) If we determine that cascading rewrites need to be supported, we can make a rehandle directive which is exactly the same as rewrite but allows cascading. I have not yet found a situation where rehandling in the Caddyfile is the only way to an elegant solution.

Example:

rehandle /docs/* /foobar

5) Other rewriting directives are not mutually exclusive. There are other directives like strip_prefix, strip_suffix, and uri_replace which change the request URI like rewrite does, but these directives are not mutually exclusive (with each other and rewrite) because they do not signify the same intent as rewrite. This took forever for me to figure out: the rewrite directive says "change the URI to this, as if the request came in like that" but the other directives say "manipulate the URI a bit before further processing". The rewrite directive will still be able to do these things like stripping prefix/suffix and do replacements, but as subdirectives.

Example: both of these will be evaluated:

strip_prefix * /foo
strip_suffix .html

6) Implement a route directive which takes a matcher and then opens a block of directives. Any directives contained within it are evaluated in the order they are specified. I don't think new matchers can be defined in the block... in fact, only handler directives should be able to be used, I think; the whole block is treated as a single unit (but matchers can be used with each individual directive like normal) when composing handler routes. The route directive will be evaluated after the middleware handlers but before the terminating handlers like file_server, reverse_proxy, and php_fastcgi.

For example, to solve this problem, we can reduce its solution to about 4 lines with an explicit route:

rewrite /titi* /toto.png
route {
    file_server /toto.png
    redir https://anotherwebsite.com
}

7) Change the matcher syntax from matcher name { ... } and match:... to @name (or =name, not sure yet) for both definition and usage.

Example (somewhat contrived): Before:

matcher phpFiles {
    path *.php
}
reverse_proxy match:phpFiles localhost:7777

After:

@phpFiles {
    path *.php
}
reverse_proxy @phpFiles localhost:7777

8) Remove special env variable parsing from Caddyfile parser. It is no longer needed in v2 and we can use the new, standard placeholder system. https://caddy.community/t/caddyfile-and-placeholders/6732?u=matt

Feedback is welcomed, but I'm going to start working on these changes tomorrow.

[mholt]

Items 1, 2, and 7 are finished, 4 and 5 are more like notes/clarifications, and 8 is in progress... so that really leaves 3 and 6 as the major TODOs yet.

For 3, I'm thinking instead of a table format under a single rewrite directive, for rewrites which should be mutually exclusive:

rewrite {
    @matcher1 /foo
    @matcher2 /bar
}

Something like that.

[smebberson]

Will this work also include allowing the Caddyfile format to support directives in https://caddyserver.com/docs/json/apps/tls/automation/policies/management/acme/ and https://caddyserver.com/docs/json/logging/?

[mholt]

@smebberson Not specifically this issue, but the Caddyfile upgrades as a whole, that are planned before the release candidates, yes

[smebberson]

I did actually discover https://github.com/caddyserver/caddy/tree/v2#how-do-i-avoid-lets-encrypt-rate-limits-with-caddy-2 just recently (I haven't tried it yet though), so maybe more of that capability exists than is written in the docs at the moment.

[mholt]

I decided to leave rewrites be mutually exclusive, without a single table as in https://github.com/caddyserver/caddy/issues/2959#issuecomment-571395901.

These upgrades are done! (Logging, error handling, TLS stuff will all come later.)

Please try it out as soon as you can, before the next beta release, which will be in a few days I think.

[sarge]

@mholt would you be able to provide an example of a reverse_proxy + stripping the prefix on a particular route.

I have not quite groked how that will work.

/users route {
    reverse_proxy https://upstream:8080
    strip_prefix /users
}

[mholt]

@sarge

route /users* {
    strip_prefix /users
    reverse_proxy https://upstream:8080
}

Directive name always comes first. Within a route, directives are ordered by appearance.

FYI, there might be some more changes coming up, after a discussion with a user tonight with a common use case that I don't think we've covered yet.