Starlette: Caching backends

[argaen]

Hello! I think it would be interesting having a section in Starlette docs documenting how to use caching as this is something you end up doing typically for lots of services.

I'm the maintainer of aiocache which is a caching framework agnostic to any http framework. I could write docs about things like:

• how to use aiocache with starlette for caching endpoints
• how to write configuration for caching and load it when starting a Starlette app

Any other thing you could think of you usually do when setting caching for your app. Also happy to receive comments, issues and PRs if you think there is anything missing to integrate nicely with Starlette :).

Would you be interested on me providing a PR documenting that? A new section called "Caching" in https://www.starlette.io/ maybe?

[tomchristie]

Amazing work. You've basically nailed that one.
I'm not really sure about the best way to go about documenting it.

Either we could create a "Plugins" section and defer to your package for the main documentation,
or we could create a "Caching" section. (Which makes sense inasmuch as its clearly a reasonable topic to have a section on, but is also slightly odd, in that we'll basically just say "use this other package, here".)

I guess one other option would be to include a CacheMiddleware that uses aiocache and that implements HTTP caching, based on the response Vary and cache control headers.

[tomchristie]

If we did just include a Caching section directly, then I think one useful thing in particular would be demonstrating how to integrate it with the rest of the configuration https://www.starlette.io/config/

Also, a thought, would it make sense to be able to instantiate the cache from a CACHE_URL similar to how the database URLs work in configuration?

app.py:

from aiocache import Cache
from starlette.config import Config

config = Config(".env")

DEBUG = config("DEBUG", cast=bool)
DATABASE_URL = config("DATABASE_URL")
CACHE_URL = config("CACHE_URL")

cache = Cache(CACHE_URL)

.env:

DEBUG=true
DATABASE_URL=postgresql://localhost/exampledb
CACHE_URL=redis://localhost:6379/0

[argaen]

Amazing work. You've basically nailed that one.

thanks :)

Either we could create a "Plugins" section and defer to your package for the main documentation,
or we could create a "Caching" section. (Which makes sense inasmuch as its clearly a reasonable topic to have a section on, but is also slightly odd, in that we'll basically just say "use this other package, here".)

Yes, I also have mixed feelings on creating a "Caching" section and then referencing directly to another package... Happy to go with the "Plugins" section. I have no strong opinion there so I'll let you decide since you are the maintainer :P.

Maybe the "Caching" section and a big disclaimer in the saying its using an external package in this section and examples? Not sure though...

[argaen]

If we did just include a Caching section directly, then I think one useful thing in particular would be demonstrating how to integrate it with the rest of the configuration https://www.starlette.io/config/

Yes definitely, this is the second item in the list I put :P

Also, a thought, would it make sense to be able to instantiate the cache from a CACHE_URL similar to how the database URLs work in configuration?

Hmmm interesting... Although I'm not 100% sure on adding the Cache type since there is a caches.create already and could end up confusing people on when to use one or the other. Wouldn't mind adding support for it in caches.create though

EDIT: I also have to explore in detail how Config for Starlette works to make up my mind on what is the best way to write an example integrating the config and the caching :)

[tomchristie]

this is the second item in the list I put

Hah, yes so it is.

[tomchristie]

Thinking aloud snippet:

# Default to in-memory SQLite database and in-memory cache for lazy development.
DATABASE_URL = config("DATABASE_URL", default="sqlite://")
CACHE_URL = config("CACHE_URL", default="memory://")

[tomchristie]

EDIT: I also have to explore in detail how Config for Starlette works to make up my mind on what is the best way to write an example integrating the config and the caching :)

Well, I'm pretty convinced that cache URLs would be a really nice way to go for most cases.

For more complex stuff you'd use the more complete configuration style that you provide, and just populate bits like the hostname (or anything else that varies across deployment environments) from the config.

```python
caches.set_config({
'default': {
'cache': "aiocache.RedisCache",
'endpoint': config('REDIS_HOSTNAME'),
'port': config('REDIS_PORT', cast=int, default= 6379),
...
}
})

[tomchristie]

I'm kinda minded towards us including it in the docs, since I think it'd guide users down the right path. (Tho I'm not 100% yet)

Personally I'm typically most interested in using caching at a reasonably low-level, such as for caching particularly frequently used or expensive database lookups. However there's also two cases I can see where we could consider including built-in support that integrates with aiocache...

1. Site-wide HTTP cache middleware. (Given that it'd be ASGI middleware, it'd be reusable across any other ASGI frameworks too)
2. Per-page HTTP cache endpoint decorator.

Have you seen any integrations that do either of those with Sanic, aiohttp or anything else? Are either of those things that you've considered?

[tomchristie]

One other thing that occurs. Now that we have a DatabaseMiddleware, we're able to consider supporting different kinds of session backends. (We currently have a signed cookie backend, but it'd be really nice if we had options for signed cookie backend, database backend, and cache backend.)

Being able to easily use redis or memcache for the session backend would be a really nice plus.

[argaen]

Have you seen any integrations that do either of those with Sanic, aiohttp or anything else? Are either of those things that you've considered?

About use cases, in production services (we use aiohttp) we mostly do as you say, low level function accessing slow services and query calls.

Regarding the middleware in our use case we rely in nginx doing this but in case there is nothing in front of the app, it's also a valid use case.

About caching directly the endpoint I haven't seen this use case but because it's hard to cache the whole endpoint but again, it may be a valid for some simple endpoints

[tomchristie]

Okay, so...

I think it’d be worth having in the main docs, if there’s an API for instantiating a cache instance from a resource URL. That’d fit in really nicely with everything else, and would give us an implementation against which we could provide a CacheSessionBackend.

Also: Is connection pooling transparently supported? Could it be? Should it be?

[argaen]

if there’s an API for instantiating a cache instance from a resource URL

sounds reasonable, I'll add the feature in aiocache and update here once its supported/released. May ask you to review it before that

Is connection pooling transparently supported? Could it be? Should it be?

What do you mean by transparently supported? Both mecached and redis implementations use the pool provide by the clients
https://github.com/argaen/aiocache/blob/master/aiocache/backends/memcached.py#L15
https://github.com/argaen/aiocache/blob/master/aiocache/backends/redis.py#L217

[tomchristie]

sounds reasonable, I'll add the feature in aiocache and update here once its supported/released. May ask you to review it before that

Sure thing!

What do you mean by transparently supported? Both mecached and redis implementations use the pool provide by the clients

Great! (Transparent, in that you'll get a connection pool rather than a single connection, without having to explicitly ask for the pool first.)

One other question: Have you considered having the package housed under a GitHub organisation, such as aio-libs? (I see you're a member of that org)

[argaen]

Great! (Transparent, in that you'll get a connection pool rather than a single connection, without having to explicitly ask for the pool first.)

Yes, you always get a connection pool

One other question: Have you considered having the package housed under a GitHub organisation, such as aio-libs? (I see you're a member of that org)

Good point yep :). I asked for it in the beginning but the library wasn't mature so was told to ask again in the future but never did. I don't have strong opinions on this but its true from consumer perspective it would be safer if it would be inside an organisation and ideally with multiple maintainers. @asvetlov what do you think? Also happy to move it to any other organisation

[argaen]

@tomchristie here is the proposal: https://github.com/argaen/aiocache/pull/432. Let me know what you think, happy to do further changes.

[argaen]

Coolio, I still want to do some couple of things before next release but I would say in 1-2 weeks there will be a new one including that.

So... do we go with creating the Caching section in the docs? Is there a specific structure or sections you would like to have in there? If you haven't though about it yet I can start with a proposal and then evolve from there, but all ears if you have anything

[tomchristie]

do we go with creating the Caching section in the docs?

Exactly, yup. I figure just low level get/set API rather than per page caching or anything like that.

[argaen]

Hey, so I've started with this playing around and I've found we can't use the decorator directly with the views because the request object is different sometimes. Code I've used:

import asyncio
from aiocache import cached
from aiocache import SimpleMemoryCache

from starlette.applications import Starlette
from starlette.responses import JSONResponse
import uvicorn


app = Starlette(debug=True)


@app.route('/')
@cached(ttl=20)
async def homepage(request):
    print(SimpleMemoryCache._cache)
    await asyncio.sleep(5)
    return JSONResponse({'hello': 'world'})

if __name__ == '__main__':
    uvicorn.run(app, host='0.0.0.0', port=8000)

and the output doing multiple requests (when there are multiple INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200 it means it hit the cache, when its printing the cache it didn't:

{}
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
{'__main__homepage(<starlette.requests.Request object at 0x110f3b320>,)[]': <starlette.responses.JSONResponse object at 0x110f49748>}
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
{'__main__homepage(<starlette.requests.Request object at 0x110f3b320>,)[]': <starlette.responses.JSONResponse object at 0x110f49748>, '__main__homepage(<starlette.requests.Request object at 0x110f3b2e8>,)[]': <starlette.responses.JSONResponse object at 0x110f3b390>}
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
{'__main__homepage(<starlette.requests.Request object at 0x110f3b2e8>,)[]': <starlette.responses.JSONResponse object at 0x110f3b390>, '__main__homepage(<starlette.requests.Request object at 0x110f49a90>,)[]': <starlette.responses.JSONResponse object at 0x110f49a20>}
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
{'__main__homepage(<starlette.requests.Request object at 0x110f49a90>,)[]': <starlette.responses.JSONResponse object at 0x110f49a20>, '__main__homepage(<starlette.requests.Request object at 0x110f49b70>,)[]': <starlette.responses.JSONResponse object at 0x110f3b390>}
INFO: ('127.0.0.1', 61045) - "GET / HTTP/1.1" 200
{'__main__homepage(<starlette.requests.Request object at 0x110f49a90>,)[]': <starlette.responses.JSONResponse object at 0x110f49a20>, '__main__homepage(<starlette.requests.Request object at 0x110f49b70>,)[]': <starlette.responses.JSONResponse object at 0x110f3b390>, '__main__homepage(<starlette.requests.Request object at 0x110f49748>,)[]': <starlette.responses.JSONResponse object at 0x110f49860>}

This is not a big issue because you can cache inner functions but it may confuse users caching functions that receive the request object. I see two options:

• We put a big disclaimer in the docs saying not to cache functions receiving the request object
• We implement the __eq__ in the request object so if the requests are equivalent, they are equal (I think that may be complicated though...)

What do you think?

[tomchristie]

I'm not sure I'd use a decorator in that way in many uses cases - if there's any aspect of the page that varies by user-authentication, then it's not going to work. Really you want HTTP-level caching for that kind of functionality.

I'm also not quite sure what you mean when you describe the behavior. We could add an __eq__ to Request - They ought to match, so long as the underlying scope matches. The scope represents the actual state, and the Request instance is just an interface onto that state. - Even so it's not exactly true, since they might have a different request body, which you can't know until you read from the request body.

[argaen]

I'm not sure I'd use a decorator in that way in many uses cases - if there's any aspect of the page that varies by user-authentication, then it's not going to work. Really you want HTTP-level caching for that kind of functionality.

yep, that's what I thought and why I think its complex to support this, but not sure if passing the whole request object all the way is a common pattern for users.

I'm also not quite sure what you mean when you describe the behavior. We could add an __eq__ to Request [...]

yeh not 100% comfortable with this though, there are many things involved in a request that can make it different. I'll go with an example showing how to cache an expensive function call and an http caching middleware plus how to link this with the config. Once I'll have it we can discuss a bit more with real examples :)

[florimondmanca]

Hi there,

I just created the asgi-caches repo to experiment with caching middleware and decorators for ASGI apps (no code for now, only docs). Basically a remake of Django's cache utilities based on async-caches. The proposed API is somewhat close to what can be seen further up this thread:

import math
from caches import Cache
from starlette.applications import Starlette
from starlette.endpoints import HTTPEndpoint
from starlette.responses import JSONResponse
from asgi_caches.middleware import CacheMiddleware
from asgi_caches.decorators import cached

app = Starlette()

# Setup a cache (in-memory).
cache = Cache("locmem://null", key_prefix="my-app", ttl=2 * 60)
app.add_event_handler("startup", cache.connect)
app.add_event_handler("shutdown", cache.disconnect)

# Add application-wide caching.
app.add_middleware(CacheMiddleware, cache=cache)


# This is route is cached for 2 minutes
@app.route("/")
async def home(request):
    return JSONResponse({"message": "Hello, world!"})


@app.route("/pi")
@cached(cache, ttl=None)  # Cache forever
class Pi(HTTPEndpoint):
    async def get(self, request):
        return JSONResponse({"value": math.pi})

The goal is to achieve framework agnosticity, and stick as close as possible to HTTP standards.

I anticipate that Django's implementation will help a lot with figuring out the edgy bits. In particular, the computation of the cache key will be based on MD5 hash computed from the request method, URL and headers.

Any feedback welcome! 😄

[patrys]

@florimondmanca Have you seen https://github.com/rafalp/async-caches?

[florimondmanca]

@patrys Yup, and if you look closely you'll see that async-caches is a dependency for asgi-caches. I initially thought of adding the ASGI functionality to async-caches via a PR, but figured that the nice frontier between the generic async world (async-caches) and the ASGI world (asgi-caches) would make things easier to maintain and iterate. As with many third-party ASGI libraries, there is room on my side for eventually merging it into Starlette, provided built-in caching support in Starlette is on the roadmap.

[cancan101]

@florimondmanca, I see that https://github.com/florimondmanca/asgi-caches is now archived. Do you have any suggestions for replacements / alternatives to it?

[florimondmanca]

@cancan101 Hi, nope, don't know of any equivalents or alternatives for now.

[cancan101]

@florimondmanca , gotcha. And what was the reason to archive the repos? Is the code useable?

[florimondmanca]

I archived the repo mainly because I didn't have time / interest in working on it anymore, at a time when I was going places on the number of projects I was maintaining.

But doesn't mean there's nothing worth anyone's interest there. 😄 My approach was to take inspiration from Django's implementation and ramp up from there. I don't remember if the code was in a usable state yet (you'd need to check any tests), but eventually it could have become something.

Anyone welcome to steal any part of it, obviously!