Caddy: Cache middleware

[mholt]

Caddy needs a cache middleware.

I'll be working on something basic this week. Probably just a basic LRU replacement algorithm and a configurable expiration time. In fact, if the gain is significant in nearly all cases, maybe it should be enabled by default... (a rare exception to the rule that all middleware need to be enabled explicitly using the Caddyfile).

[thomas4019]

I have a pull request with the beginnings of a solution here, https://github.com/mholt/caddy/pull/26

[mholt]

For anyone who is reading this thread who would like to contribute and is interested in cache middleware, feel free to pick it up - see the PR issue for details.

[coocood]

I just came across this thread and wondering maybe https://github.com/coocood/freecache can be used as cache middleware.

[mholt]

Perhaps an easier start to this middleware would be to implement some basic caching headers like ETag, Expires, Last-Modified, etc. See #191 for possible related discussion.

[hackintoshrao]

There is certainly a need for caching to scale Caddy. The profiling shows that the performance is hindered by File I/O being made for every request.Will soon release documents and reviews on performance and enhancement suggestion for each middleware from the profiling and performance tests I'm doing. I'm also looking at Groupcache and checking out whether it could be integrated for caching static contents

[tgulacsi]

Using go-search.org, I've found two candidates to take parts from: https://github.com/gregjones/httpcache and https://github.com/lox/httpcache .

My idea is to start small: have an in-memory response cache, with some nice eviction algorithm (W-TinyLFU, implemented by Damian Gryski: github.com/dgryski/go-tinylfu).
What we have to avoid at all cost, is caching something that shouldn't be - so take the header parsing logics from https://github.com/lox/httpcache and decide whether to cache and whether to serve from the cache, conservatively.

Maybe the "start small" method should be taken seriously: start with an in-memory TinyLFU, and cache only unambigous resources: file handlers, static file's contents.

Those static file contents could use fsnotify to avoid statting them! But I think that should be a separate PR.

[joeblew99]

I am needing basic caching for a production system.
Basic pragma tag technique would be enough for now.
I do need to be able to change the caching amount, without restarting caddy too if possible. This is because when the cache is invalidated varies by when we pump new data into the backend.
Any updates on this ?

[jupiter]

https://github.com/lox/httpcache seems almost perfect to me, if only if it weren't for this issue and that it doesn't limit the size of the cache at the moment. Perhaps we should see if we can sort that out first.

[lox]

I'm happy to collaborate with you guys, I've recently been working on splitting out just the caching logic into a standalone library, as I think it's a separate problem to actually storing and looking up cached resources. Whether you guys would use that (called cachecontrol) or httpcache is up to you.

[mholt]

Pinging @weingart since interest in a cache middleware seems to be increasing. :smile:

[jordic]

I'm just started thinking on a cache directive, something simple like:

cache / 1h
or
cache / 1h { except /wp-login /wp-admin expire-key cache_delete }

What i think is a good starting point is:
https://www.nginx.com/blog/nginx-caching-guide/

Can Benefit

• proxy directive
• fastcgi
• futre minifys or html compressors.

Points:

• On initial I think is better to focus in only caching GET and HEAD requests, and I think only for text content/types like (html, json, xml... ). Is absurd to cache binary files already served from filesystem. Actually the stdlib http server is good serving static files.
• A Filesystem backend is enough. The general idea is to cache response blobs to files, and serve them directly with the send_file ( fileserver in go works this way), this way the memory managament is done by the kernel, and simplifies the problem. I can elaborate more on this, but I think that going forward on it, is out of scope of a web server (There are other good tools around to use just in case)
• The cache directive should be defined per virtualhost. And should be up on the middleware chain, as starting, I think it should go just after the prometheus ...
• The main principle for operation is to have a map[path]cacheitem, and on an incoming request, just check if key exists. In case it exists. The cacheitem should contain an expiration time, and a md5 of the host + path ( used to store the file on disk ). If item is not expired, it should serve directly the file.
• If key doesn't exists or is expired and must be cached (is not on the exclude list), we need to record the response, store it on disk, and update the map.
• We can also have some kind of manually expire mechanism ( perhaps a request with a key ) and on this case, we have to iterate over the dict, and delete matching keys. I think, something as simple as strings.HasPrefix is enough for this case, if you want to delete entries on path /posts .. (we can iterate over the map, and delete all /post (Not sure if is better to delete them or to mark them as expired, with a expired date)
• Cache operations on files should be atomic. (Write first to a temp file, and move later to the correct key index? (Or we need some kind of locks on files) We don't want to serve a item from cache that is still not updated, also we don't want to duplicate efforts storing cache items in concurrent requests.

Doubts and things to solve.

• I don't know what we should do with headers for cached requests.

What you think?
@mholt @weingart @lox @jupiter

[jupiter]

@lox That would be awesome.

[weingart]

I'm sorry that this has taken so long for me to respond. Let me separate
out a few different types of caching:

• using various http headers to encourage (or discourage) clients and
  cache servers to cache content. Such as cache-control, etc. This is a
  pretty decent intro:
  http://www.mobify.com/blog/beginners-guide-to-http-cache-headers/
• caching things by writing them to disk, usually by being a proxy to a
  backend of some sort. In the end, this still causes I/O to disk, possibly
  optimized via sendfile(2) or other OS dependant methods.
• caching small static content (such as favicon.ico, etc) in memory, such
  that no I/O hits disk at all (other than swap, if you use it, please
  don't!).

At one point, I was attempting to do #1 and #3 in the same shot. I've
backed off, and am implementing #1 separately. The only real reason that
it sort of requires some middleware, is that older browsers are more
finicky about how you tell them about content expiry.

The other half, #3, really needs to be included in the fileserver.go
(currently non-middleware) section. I personally wish for this, so that I
can minimize the amount of I/O I incur, as "in the cloud", I/O costs money,
but I already have the memory, and serving from memory with a timeout for
the content being cached is faster and easier on the wallet. :)

Number 2. That one, I see so much trouble in. I personally am not so sure
that it really belongs in a general purpose http server. I would point at
squid, or other such, purpose built solutions for this. Note, I don't with
to discourage people that want to implement such a beast, but I really do
think that a serious http based caching service should really be
implemented separately. If you're really keen, using something like
groupcache, implementing a possibly distributed caching system, that can
front caddy (or possibly other http style servers), would be a large
project in itself as well.

Thoughts?

-Toby.

On Mon, Apr 4, 2016 at 12:47 AM, Pieter Raubenheimer <
[email protected]> wrote:

@lox https://github.com/lox That would be awesome.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/mholt/caddy/issues/10#issuecomment-205178125

[jordic]

For 1)
Separate 1 sound as the way to go. Also this should be the first part. For example, is necessary to respond with etags (I think this is already on tip)

For 2)
I think we need to focus on a target use case. We don't target users that are using varnish, or squid, or something else. As nginx, to have a small file based cache directive, is enough. There are a lot of situations where you can benefit from having it. Imagine a traffic pike on your wordpress site.. Just update the caddyfile and you are fine! Or imagine a busy backend, where you can't handle all the petitions, you can cache for an small time, and reduce notably these...

At the beginning, when we started this issue, and asked others (on the gophers slack), the common opinion is to leave the cache feature to the kernel, using the send_file. Is the same is doing nginx, and also I think is what is doing varnish.

By the way, I think that the cache storage should be decoupled from the cache middleware... At starting, it can be file based, but later, a redis backend could be implemented (also, groupcache, memcached)..

For 3)
Caching small files that can be served from disk is not a benefit. On the end, if you really need to handle big loads for static files, better you put them on a CDN, or object storage like (S3, gcloud, etc..). The cache directive must focus, on caching responses from expensive backends ( proxy, php_fpm, anewcoolmodulethatcanminifyhtml, etc... ) that is the most common bottleneck to your app.

[lox]

Out of interest, what value does 1 offer without actually doing any intermediary caching in the server?

[mholt]

@lox

Out of interest, what value does 1 offer without actually doing any intermediary caching in the server?

A client can cache resources so it doesn't even have to ask the server for them, so the headers can enable client-side caching and inform intermediaries what to or not to cache. (Cache-Control! That should definitely be supported.)

@weingart Thanks for your brain dump :smile: I do largely agree with your points as well as what @jordic said; caching is not a light/easy task. I think Caddy should support basic caching features in its core, and leave advanced / power-user caching to optional plugins, especially if they involve third-party software/dependencies.

Tobias, what parts of this are you working on so far? It sounds like 1 and 3; but what is the caching strategy you are implementing?

[weingart]

I'm currently working on #1 (expiry headers, cache-control, etc). The
other two I really think should be provided by 3rd party middleware. I'm
also working on #3, but in part, that one is likely to be more
controversial, in that the cleanest way to implement (that I can see) is to
move the current fileserver.go into middleware, where it can still be
instantiated as an anchor, but could possibly be replaced by something that
does #3.

I think for immediate concern is #1, to be able to more easily specify
content expiry headers in a static configuration manner. I consider #1 to
be essential, #3 to be an experiment I'm working on for myself, and #2 a
significant, non-trivial, endeavour.

A side note, the caching layer as described in #2, would make caddy now
have a "write to disk" style of code. I'd consider this a possible
security risk/vector. Yes, I understand that the user/content only has
minimal control over the location that things are written in (hash or hmac
values, oh, please use hmac instead of hashes, it allows for the
invalidation of the cache in a quick manner). If nothing else, #2
incorporates the creation and management of persistent state, which is
notoriously difficult to deal with (usually).

-Toby.

On Thu, Apr 7, 2016 at 10:38 PM, Matt Holt [email protected] wrote:

@lox https://github.com/lox

Out of interest, what value does 1 offer without actually doing any
intermediary caching in the server?

A client can cache resources so it doesn't even have to ask the server for
them, so the headers can enable client-side caching and inform
intermediaries what to or not to cache. (Cache-Control! That should
definitely be supported.)

@weingart https://github.com/weingart Thanks for your brain dump [image:
:smile:] I do largely agree with your points as well as what @jordic
https://github.com/jordic said; caching is not a light/easy task. I
think Caddy should support basic caching features in its core, and leave
advanced / power-user caching to optional plugins, especially if they
involve third-party software/dependencies.

Tobias, what parts of this are you working on so far? It sounds like 1 and
3; but what is the caching strategy you are implementing?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/mholt/caddy/issues/10#issuecomment-207223944

[lox]

@mholt fairly familiar with what the caching headers are actually for (via https://github.com/lox/httpcache), trying to figure out what an http server that wasn't using the headers to actually cache/store/validate would need beyond "Add-Header" for different types of content.

@weingart out of interest, how does hmac allow for invalidation of cache in a quick manner vs non-cryptographic hashes? We are using sha256 in httpcache, but are likely going to move to fnv64a in the next release.

As I mentioned above, I'm currently extracting the validation and transport piece out of httpcache into a dedicated golang library. I've put my WIP up at https://github.com/lox/cachecontrol. Basically I want a library that is entirely BYO persistence layer. Interested in whether this will be a useful thing for you guys.

[weingart]

By changing the key, the hmac will generate a different hash. In this way,
you can invalidate the whole cache (well, make it disappear) quickly.

-Toby.

On Fri, Apr 8, 2016 at 11:37 PM, Lachlan Donald [email protected]
wrote:

@mholt https://github.com/mholt fairly familiar with what the caching
headers are actually for (via https://github.com/lox/httpcache), trying
to figure out what an http server that wasn't using the headers to actually
cache/store/validate would need beyond "Add-Header" for different types of
content.

@weingart https://github.com/weingart out of interest, how does hmac
allow for invalidation of cache in a quick manner vs non-cryptographic
hashes? We are using sha256 in httpcache, but are likely going to move to
fnv64a in the next release.

As I mentioned above, I'm currently extracting the validation and
transport piece out of httpcache into a dedicated golang library. I've put
my WIP up at https://github.com/lox/cachecontrol. Basically I want a
library that is entirely BYO persistence layer. Interested in whether this
will be a useful thing for you guys.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/mholt/caddy/issues/10#issuecomment-207727967

[jordic]

@weingart
I don't see the point about 3).
filserver is a middleware... but is the last on the middleware onion, as I understand here:
https://github.com/mholt/caddy/blob/master/server/virtualhost.go#L30
From a middleware, you can intercept any request, and respond to it.. (using or not the subsequent layers): test if path exists on your backend, and serve with stored contents...

Ok, but, how to store items on the cache?: Build a ResponseRecorder, (execute the rest of the layers with this one, and you will have access to the written bytes blob. (@mholt, is this correct?)
Anyway, if what you want is serve files from ram.. just mount a tmpfs, and serve from them (let the OS do their job).

For the state:
I think a caddy new start should bring us a clean cache.
For the rest, i will preserve state in memory. As explained with a safe map. And rebuild each entry as needed. Expiring all, is as easy as initializing the map. Expire a group of keys, just iterate over the map. and clean the keys ( no need to clean files on disk, they will be overwritten by new entries when arrived). Check if a key exists on the map, then check if key is expired..

Finally, I see one more important point on the main cache func... The rules to make a url eligible to be cached.

Incomming request.

1. Cache middleware test if url is cached, if true and not expired, serve cached content. (END)
2. If not cached, test if url is elegible for being cached. If not, continue with the middleware onion.
3. If true, start request recorder, record and store the contents.

[lox]

You will want to stream the response body to the cache at the same time as to the client, blocking the downstream client until a response is cached causes timeouts for large files. Ideally concurrent requests to a resource being cached should be deduplicated too, but that is harder to do without a disk buffer.

On 10 Apr 2016, at 2:26 PM, Jordi Collell [email protected] wrote:

@weingart
I don't see the point about 3).
filserver is a middleware... but is the last on the middleware onion, as I understand here:
https://github.com/mholt/caddy/blob/master/server/virtualhost.go#L30
From a middleware, you can intercept any request, and respond to it.. (using or not the subsequent layers): test if path exists on your backend, and serve with stored contents...

Ok, but, how to store items on the cache?: Build a ResponseRecorder, (execute the rest of the layers with this one, and you will have access to the written bytes blob. (@mholt, is this correct?)
Anyway, if what you want is serve files from ram.. just mount a tmpfs, and serve from them (let the OS do their job).

For the state:
I think a caddy new start should bring us a clean cache.
For the rest, i will preserve state in memory. As explained with a safe map. And rebuild each entry as needed. Expiring all, is as easy as initializing the map. Expire a group of keys, just iterate over the map. and clean the keys ( no need to clean files on disk, they will be overwritten by new entries when arrived). Check if a key exists on the map, then check if key is expired..

Finally, I see one more important point on the main cache func... The rules to make a url eligible to be cached.

Incomming request.

1. Cache middleware test if url is cached, if true and not expired, serve cached content. (END)
2. If not cached, test if url is elegible for being cached. If not, continue with the middleware onion.
3. If true, start request recorder, record and store the contents.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[jordic]

Good point for the first!
Not sure, but perhaps a pipe from the upstream writting the same time to disk and to downstream.

For the second, (when concurrent requests to the same resource).. two options come to my mind:
A) Lock them till resource is cached...
B) Bypass to the upstream not caching at all ...

[kennylevinsen]

@weingart I'm not sure I agree that solution 3, in-memory caching, would belong in the file server. One problem I see with this is accept-encoding. I would find storing gzip'd responses to be a high priority, and for this, a higher-level middleware would be required. If you want some rough numbers on why serving gzip'd content cached, in my opinion much more so than serving plain files from cache, see the table at https://kl.wtf/posts/minihttp/ (ignore the 39 seconds run for Caddy, as this is due to excessive GC pressure. With a bigger box, this is the same as "minihttp, disk, gzip" - the important aspect is that, at least for this test, that gzipping on the fly is expensive, and that serving a file from memory or from disk did not really show any significant difference - and that's despite _not_ using send_file).

The operation of a higher-level middleware would be a bit more complicated. If you are already progressing with injecting correct cache headers, the cache middle-ware would in theory consume these and act as a CDN-style cache proxy. If the cache-header "generator" generated a full set of cache-headers, which include different policies for CDN's and client-side caches, then this could in theory work, and would also include the benefit of automatically handling CDN's correctly (which you generally want to cache things differently than the client if there is dynamic content involved). Likewise, dynamically generated, but not _really_ dynamic, such a a markdown blog, will also be cached.

At this level, we would then store a response for each type of Content-Encoding. We could obey Vary to make things fancier, but the problem mainly lies in just having plain and gzip'd content.

The alternative would be some potentially complicated rules in a Caddyfile describing cache behaviour. A simple version would be "cache 2h /blog", but with dynamic content, these rules might turn less amusing, and partially duplicate the work done to ensure proper cache-headers.

I personally like the thought of the "cache-obeying" variant better, so that cache settings just relate to headers, and middleware will be able to say "DON'T CACHE THIS" (dunnnnn dun dun dun...). There are some complexities in handling things such as query strings, but it's doable - some might require a few tunable settings.

As for to disk, I see this to be a very minor consideration, in the sense that the cache store could be an interface that could easily be swapped between memory and disk, maintaining the cache logic. In other words, it can be decided on _after_ the caching mechanism has been established, which is the hard part.

The previously mentioned lox/httpcache project seems to be close. The issue is has with pausing streaming of the response should be fairly easy to fix using a new ResponseWriter that uses a io.MultiWriter to write down/upstream (not sure which way's up in this chain), with a bytes.Buffer as a secondary writer to cache on the fly. Deduplication of multiple requests will be a bit tricky - it might be easier to conclude "Oh, the cache entry is unavailable - let's serve things straight then" until caching is complete.

Thoughts?

[lox]

I've gotten https://github.com/lox/cachecontrol to the point where it captures most of the core rules around caching from rfc7234. Basically it should be useful for determining if a response is fresh or whether it needs validation.

[flibustenet]

Hi,

With nginx I use this cache method:
When the file exists on disk, nginx serve the file like any static file.
When the file doesn't exist it call the app (proxy). The app send the page and write it to disk. When the app decide to refresh the file it just need to delete the file.
http://nginx.org/en/docs/http/ngx_http_core_module.html#try_files
Don't know if it's currently possible with nginx ? And if not maybe it could be added in the cache middleware or elsewhere (a new issue for this) ?

[jlsjonas]

Hey

I was wondering how we're doing on adding the caching headers (in a similar way to what @jordic described)

@weingart do you have your progress up somewhere? maybe I can pitch in to progress it, since caddy feels awesome; but I'd rather have http-server-level (basic) cache control

[lox]

So any reason to not just use my httpcache library? It's exactly what it's designed for, would be useful to get some feedback on how it might better suit the usecase y'all have.

[mholt]

@lox I'm certainly not opposed to it (the lack of size constraints is a little worrisome, but I'm not sure what the implications there are) -- the thing about any cache implementation is that it absolutely has to be benchmarked properly to ensure that it is in fact delivering a speed boost within certain constraints of memory and disk space.

[ghost]

I have a suggestion for a cache library
https://github.com/golang/groupcache

I don't have intimate knowledge of the requirements for this present issue. However, the architecture, functionality and support is high quality IMHO.

[lox]

@gedw99 groupcache has no expiration mechanism, it's for immutable content.

[lox]

@mholt yup, agreed. I'd say that compliance with the spec is pretty important too though, and it's more complicated than first glance suggests. Happy to work with you guys to get size constrained storage and high performance. I've been working on a rewrite of httpcache that uses the cachecontrol library I wrote, should end up much leaner and quicker.

[joeblew99]

Makes sense.

[tomByrer]

Any progress on this?

[lsferreira42]

Yes, i've been using caddy for some time now and this is a real necessity.
I want to help on that, there is already a PR from Thomas but someone made some progress on that?

[mholt]

Based on the discussion since that time, I'd recommend starting a new plugin from scratch that meets a very specific caching need, complete with carefully controlled benchmarks.

[mholt]

Just found this from a post on HN: https://github.com/nicolasazrak/caddy-cache - still WIP. Looks interesting.

[lox]

I'll see if I can get httpcache integrated to it :)

[azjkjensen]

Any progress on this lately? What is the latest status?

[mholt]

If that middleware above, by @nicolasazrak, is useful, I'd love to see it finished and published to the Caddy download page. (New site coming in a while.)

[nicolasazrak]

@mholt when I started the project the idea was to send a pr and be published in the download page but I found that doing caching was harder than I thought. I will continue with this project now that I have some free time and try to have something more polished

[azjkjensen]

@nicolasazrak 🙌🏻🙌🏻🙌🏻

[mholt]

@nicolasazrak Sounds great. :) Yes, caching is hard. I would have attempted it myself... ;)

[azjkjensen]

@nicolasazrak let me know if I can help. I am a noob at Go, but I'd love to contribute.

[nicolasazrak]

@azjkjensen help is always appreciated, if you want send me an email so we can work together (my email is on my profile)

[vladbondarenko]

Sorry for not reading all the thread ;) but if cache will be implemented, please check if its possible to make some "shared cache" in Caddy.
Like:
If I have several caddy servers (for redundancy or stanby) they are having the same cache storage. if 1-st Caddy caches something, 2-nd Caddy can use that object from cache and not fetch from backend again.

Maybe store cache in Redis or something.

That thing is really need for big prod sites. Unfortunately Varnish dont have such, will be cool if Caddy can.

[weingart]

Sounds like you're looking for groupcache as one option...

-Toby.

On Mon, May 8, 2017 at 12:42 AM, vladbondarenko [e-mail]
wrote:

Sorry for not reading all the thread ;) but if cache will be implemented,
please check if its possible to make some "shared cache" in Caddy.
Like:
If I have several caddy servers (for redundancy or stanby) they are having
the same cache storage. if 1-st Caddy caches something, 2-nd Caddy can use
that object from cache and not fetch from backend again.

Maybe store cache in Redis or something.

That thing is really need for big prod sites. Unfortunately Varnish dont
have such, will be cool if Caddy can.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mholt/caddy/issues/10#issuecomment-299795925, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABLDSa-wW4ZancCZxtMOAYbN6bLicLsQks5r3sd4gaJpZM4EKzni
.

[mholt]

Very happy to close this with the near release of the http.cache plugin. :tada: https://github.com/nicolasazrak/caddy-cache

Thank you @nicolasazrak!