Netbox: Implement Caching Mechanism

[tyler-8]

Environment

• Python version: 3.6.7
• NetBox version: 2.4.8

Proposed Functionality

Django offers a caching mechanism to improve page load performance. The three primary methods are

• per-site cache
• per-view cache
• template fragments

Per-site is invalidated strictly by the cache expiry time - meaning if you update an object, the changes don't appear until the cache for that view/page/object expires. This isn't very useful for Netbox workflows.

Per-view and template fragments could both benefit Netbox but would require changes across the project, mostly in the Django views in each area of Netbox. This would involve some sort of CacheMixin. Django's caching interface supports several methods for cache storage (filesystem, database, memcached, redis w/plugins).

The plugin django-cacheops sets up the cache using redis and automatically hooks in to Django's signal framework for invalidating caches as objects are created/updated/deleted. This option requires very little code changes other than adding new configurations.

Use Case

Ideally a user would only need to make a few changes to the configuration in order to select the caching storage method (in the case of the per-view method) and location. From there Netbox would cache on the fly without further intervention.

Database Changes

The database or Netbox model schema would not require any changes.

External Dependencies

Depends on the path chosen

Potentially:

• redis
• memcached
• django-cacheops
• Another SQL database/table (for cache storage)

[sdktr]

Very curious to the latency enhancements this could bring in both UI and API. Caching on the 'view' level means a 'calculated' value is stored right? Not just a [set of] database call saved, but the actual rendering of the data (like prefix hierarcy)?

[tyler-8]

@sdktr That's correct. The final rendered output is what is cached so the prefix page would see big improvements for _follow-on_ visits. The first person to visit the page after the cache expiry time would still experience the page load delay due to the IP calculations, but any additional views would be almost instant.

[jeremystretch]

Tagging this tentatively for v2.6. We'll need to do a lot of exploration and testing but it seems feasible.

[paravoid]

While implementing this and looking at all the complexities around cacheability (what can be cached, for whom, for how long, etc.) it would be cool if the @cache_control and @vary_on_cookie decorators (or equivalent functionality) were also considered. They'd help with caching both in the browser, and frontend caches (like Varnish etc.)

[jeremystretch]

@lampwins One issue I've noticed with caching is that the cache is not expired on login/logout. We probably want to wrap views with vary_on_headers() to ensure a different cache per user.

[sdktr]

Is the direction of the cache implementation focussed on GUI views only or does it speed up API responses as well?

[lampwins]

Implemented in develop-2.6

[vali-um]

I was just looking through those changes as we have a small instances of netbox running and don't really want to install redis for a functionality not needed... and i'd imagine that i'm not the only one in a similar situation.
Would it be feasible to keep redis as an option and have caching optional as well? As far as i understood the changes this wold only require minor changes.

[jeremystretch]

Redis is now a requirement of NetBox starting with v2.6, not only for caching and webhooks but for future planned features as well.