Caddy: Provide all core matchers as CEL functions

[francislavoie]

CEL is super cool and powerful, makes it easier to write complex conditional logic for request matching. We can use placeholders to pull the values from the request and write conditions with it.

One thing that would make it even better though, is if all our existing request matchers were added to the CEL context as functions.

Consider this Caddyfile (example of doing all the permutations of conditional logic):

localhost

@both {
    path /abc/*
    method GET
}
respond @both "Both"

@neither {
    not path /abc/*
    not method GET
}
respond @neither "Neither"

@not_both {
    not {
        path /abc/*
        method GET
    }
}
respond @not_both "Not both"

@either {
    not {
        not path /abc/*
        not method GET
    }
}
respond @either "Either"

This is pretty wacky. Especially the "either" case.

If we had the matchers as CEL functions, we could do something like this for each of them (not all of these make sense as expressions, but this is just an example):

localhost

@both expression `path("/abc/*") && method("GET")`
respond @both "Both"

@neither expression `!path("/abc/*") || !method("GET")`
respond @neither "Neither"

@not_both expression `!path("/abc/*") && !method("GET")`
respond @not_both "Not both"

@either expression `path("/abc/*") || method("GET")`
respond @either "Either"

This reads a lot better, especially if you're comfortable with programming boolean logic.

[francislavoie]

@TristonianJones I was hoping you could clarify something - I've been digging through the CEL docs and can't figure out whether it's possible to have variadic args for function overloads. I did see that && is noted as variadic, so I assume there must be a way.

Specifically, I want to be able to support all of the following:

path(request, "/foo/*")
path(request, "/foo/*", "/bar/*")
path(request, "/foo/*", "/bar/*", "/baz/*")
...

(Note that I'll be using a regexp to expand path(args) to path(request, args) so that users don't need to specify request, which would be unnecessarily verbose in this context)

Do I just use dyn as the last type? Do I have to have users wrap args in a [] list instead?

[TristonianJones]

@francislavoie The CEL proto supports variadic arguments at an AST level, but the type checker and interpreter don't currently support variadic functions. Using a list literal as an argument is the simplest approximation to what you want: path(request, [<arg_exprs>]). Though you could also add overloads to the type-checker and interpreter for arg counts 1 .. N where N is some reasonable number to simulate what you want:

path(request, string)
path(request, string, string)
path(request, string, string, string) // and so on until you hit some reasonable limit.

If you give the overloads distinct names then the type-checker will also make it simple for the interpreter to dispatch to the correct overload at runtime: path_request_string, path_request_string_2, ..., path_request_string_20. It's currently not quite as simple as I'd like to setup the interpreter to specify both the dynamic dispatch form of the function and the specialized overload, but it's still doable and you can see an example of what I'm talking about in cel-go/ext/strings.go where I have to setup a function to handle calls to either replace or string_replace_string_int.

There's nothing really preventing us from adding support for variadic functions, but we'd need to make sure it can be specified and understood in all runtimes. For us, this is mostly just a prioritization question of which things to work on first. Recently, there have been feature requests for aliased identifiers, expression linting, and adding the var-args functions into the mix means that the CEL user-base is really maturing pretty quickly now. Exciting stuff. I hope I can get to it all soon. :)

[francislavoie]

Thanks for the answer! Sounds good.

[mholt]

Good idea! As long as these functions are only in the global scope for _request matcher CEL expressions_, then this should be fine. (As opposed to making these functions global for _all_ CEL expressions in case CEL is used anywhere else in Caddy -- the global namespace is too valuable in that case).

[francislavoie]

So I'm coming back to this to think about how it would work, I'm struggling to see how it could be done efficiently.

The problem is that matchers actually have their arguments set up on their struct, and the Match method takes only the request. That means that we would need to construct the MatchPath struct (for example) on every request and fill it with the args from the CEL function invocation, so that we can then call Match(requestFromContext) on it.

I'm don't think it's possible to pre-allocate the struct based on the compiled CEL expression to avoid creating the struct on every request...

I guess we could have all the matchers provide a struct-less Match function for the purposes of making it work for CEL, but is that worth the maintenance overhead? And if they're struct-less, how could they be loaded dynamically by their module name? 🤔

I don't know what the way forwards is here 😢

[mholt]

What if, instead, you just had access to the same properties of the request that matchers have, and you write your own "matcher"? i.e. something like request.uri.path or whatever. Oh wait, that's like {placeholders} already? (Although they get converted to strings, so we lose some type information.) We could probably make a strongly-typed request value for CEL, right?

[francislavoie]

Well @mholt the point was to get the full semantics of the existing matchers we have, like the file matcher, but be able to compose boolean logic out of them more easily than with a bunch of not matchers. You can already do stuff like path and method matchers easily in CEL because we already have the request and placeholders, but that's not enough for things that need to read from disk or whatever.

[mholt]

I understand that, and I like the idea, but I dunno how feasible it is compared to what we get from it. _Most_ of the matchers' functionality could be pretty easily exposed if we had a strongly-typed Request value, I think? And that might be easier. I'm just trying to see what is more achievable, if it's worth doing anything at all for this. But let's see if we can find a way to make existing matchers accessible, I guess.