Caddy: v2: Content negotiation

[mholt]

Caddy 2 needs a generic, powerful way to do content negotiation.

Currently, the http.handlers.encode module negotiates its encoding from the Accept-Encoding header. However, for pre-compressed assets or other kinds of content negotiation, different facilities are required.

Content negotiation can be based on MIME type (Accept header), encoding (Accept-Encoding header), or language (Accept-Language) header.

I propose a content negotiation request matcher:

"content_negotiation": {
    "mime": ["text/plain", "text/html"],
    "encoding": ["br", "gzip", "flate"],
    "language": ["en-US"]
}

For each negotiated variance, a placeholder would be added indicating what was negotiated, for example: {http.matchers.content_negotiation.mime}

TODO: We would also need a way to add a Vary header to the response. Maybe a matcher is not the right solution. This might have to be a middleware... hm.

[mholt]

As discussed in https://caddy.community/t/how-to-serve-gzipped-files-automatically-in-caddy-v2/7311?u=matt it would be useful for Caddy to be able to serve pre-compressed files as well. If enabled, this would involve using the Accept-Encoding header to see if there is a matching file on disk, and if so, serve it with the proper headers. Maybe as a guest module to the file_server handler.

[averri]

What is the workaround for implementing this in the actual version v2? Suppose the following scenario:

The browser requests the resource /js/app.js with the header accept-encoding: gzip. If the resource exists with an extension gz, like /js/app.js.gz, so the HTTP response should include the compressed file /js/app.js.gz with the appropriate response header set: content-encoding: gzip.

[polarathene]

It'd be nice for Caddy to be able to support pre-compressed brotli .br files.

Unlike gzip, brotli isn't advised for on-demand compression from what I've read, would it also be something Caddy could do if encoding was requested for static assets to compress them, but write that compressed file to disk for serving in future, or is this something that should be handled outside of Caddy? Such as a scheduled compression pass if compressible files are added to the server?(eg via user uploads)

[mholt]

It'd be nice for Caddy to be able to support pre-compressed brotli .br files.

I think so too (see my reply above)! It just needs to be discussed, designed, and implemented...

[ueffel]

An important point I'm missing here is, that the decision to use compression at all should depend on the type of data to be served. It is wasted CPU cycles to compress images, web fonts (woff2) or any other content that is already compressed.
Maybe there is a way to delay the decision to compress until the Content-Type of the response is known. So meaning the mime type of a file or the Content-Type header itself from a reverse proxied response.
Caddy then should have a way to configure the Content-Types to be compressed, something like nginx's gzip_types but not limited to one encoder but across all of them.

To pre-compression:
I think encoder modules should implement optionally an interface to provide a file extension to add to a filepath. This is then used to check for pre-compressed files. The question is which module does the checking. I'm not familiar enough with the caddy source code to make a good suggestion here. Maybe also via interface that is be implemented by modules, first of all file_server?

Next question would be how determine the order of the encodings. In #3692 I suggested a implemention of a prefered order of encodings. But this should only apply to dynamic content. The prefered order of encodings for pre compressed content may/should be different. (Brotli for pre compression because of the better compression ratio and gzip for dynamic content because of the better speed). Maybe an separate setting?
This begs a further question then: What to do first:

• Determine the encoding to be used according to the Accept-Encoding header and then check for pre compressed files of that encoding or
• Checking for pre compressed files first (all accepted encodings) and then determine the encoding according to the results

(I think as a first step pre compression should be done outside of caddy.
Transparently pre compress assets would be a nice feature but there are questions: When to do this? At startup? If requested x amount of times? As a extra step before starting caddy but built-in? How to determine when a pre compressed asset is out of date and needs to be recompressed?)