Zeronet: Prevent needing to disable HTTP Cache during development

[anoadragon453]

The browser's cache unfortunately makes development difficult as it needs to be disabled, and the browser's developer tools kept open. One small trick that can be done is append a version number to the file's import to force a refresh. For example:

<script src="my/script.js?v=1.1"></script>

However updating this version number every time an edit is made is arguably even more work for site developers... unless we could automate this behavior.

If, on page load, ZeroNet would append an auto-incrementing number to each imported resource (CSS, JS) such the unix timestamp, this problem would be solved.

Would it be possible for ZeroNet to do this processing to HTML files before serving them to the browser?

As more motivation I had a friend starting ZeroNet development today and he got stuck and confused after his site wasn't updating when he changed it. Took him an hour of frustration to eventually contact me and I told him about the cache. I know it's on our docs, but people don't always check or read that far :)

[HelloZeroNet]

it would be easier to disable the cache with http headers, but it would reduce the performance

[anoadragon453]

Perhaps disable the cache in --debug mode?

[HelloZeroNet]

I don't think that all developers runs in debug mode, so maybe --http_cache_owned_sites (disabled by default)

[anoadragon453]

My only problem with this is that it doesn't seem much different than asking people to disable the cache in the developer tools.

Is it possible to only disable the cache for owned sites?

[krixano]

In the src/Ui/UiRequest.py file, in the sendHeaders function/method, we can change the cache settings being sent to the browser (via the Cache-Control header) there.

1. I don't see how the cache will help much regarding performance. When browsers cache, they store the files on the hard drive so they don't have to get it from the internet every time. However, zites are already downloaded to the hard drive. The only thing I can think of is if either the ZeroNet client or browsers also utilize RAM caching for a session or something.
2. However, this will affect the performance of proxies. We could detect if the request is coming locally, and send different Cache-Control headers to enable caching for proxy users.
3. Images and videos are less likely to be changed. Javascript files are the most likely to be changed. I think CSS is in the middle.
4. So I propose, disable caching for js files and css files, but keep it for images and videos. And enable caching for js and css files if the request is coming from a proxy user.

[HelloZeroNet]

When I had a lower-spec machine (Core2duo) The page loads was significantly slower without caching. I suspect it's not just about the download, but it's also caches the parsing of the file.

[styromaniac]

Is it possible to have each file's cached copy expire on the checksum change of the original?

[krixano]

I was just about to edit my post suggesting that as an idea.

[krixano]

The main problem I have with the cache situation is that zites sometimes don't update unless the user clears the cache. And this becomes more important when a dev wants to get out a very important bug fix as quick as possible. We could do the whole appending version number to file name, but I think there's a better way. I think what @styromaniac proposes would be good, if possible.

[styromaniac]

Owned sites never cached + checksum versioning

[HelloZeroNet]

The resource cache is 10 minutes, so I dont think that it's really affects any, non developer user.
Adding a checksum would require constant checking of etags that makes the caching less efficient

[krixano]

I think, when a zite is updated, the client keeps track of that, and when a file is requested, it can check if it's in this array and send a no cache header

[HelloZeroNet]

The client won't request the file if it's cached, so no point to return the no cache header

[krixano]

Right.... blech

[anoadragon453]

I vote to start with disabling cache for owned sites and iterate from there?

[krixano]

The resource cache is 10 minutes, so I dont think that it's really affects any, non developer user.

I kinda disagree with this. If I have to get a very important fix out in under 10 minutes, I don't want my zite to be unusable for 10 minutes. Stuff like this has happened before.

[styromaniac]

Adding a checksum would require constant checking of etags that makes the caching less efficient

Relevant response below after I elaborated the thought behind it on riot.im

So force recache when the first call from the browser after the site update?

So, no caching on owned sites for the developer and flushing cached copies on site update for the consumer(s). This way the site isn't broken for the user for ten minutes.

Is this doable?

[HelloZeroNet]

i can add a {site_modified} variable that automatically got replaced in *.html files with the modified value in site's root content.json and with the current time if the site is owned

so you can add it to js and css files as get paramter

[anoadragon453]

That sounds simple and elegant! I'm on board.

Edit: It does still require people to do something different from normal webdev though. Maybe we can just really stress that you should add it when importing assets in all our docs and tutorials?

[styromaniac]

I know I'm kind of breaking a GitHub rule by making a comment only showing my approval, but yes, I approve.

[HelloZeroNet]

Added in latest Rev: https://github.com/HelloZeroNet/ZeroNet/commit/5af77449fad4644f0fe238893b55e6c32153ae95

Example: <script src="my/script.js?v={site_modified}"></script>

[anoadragon453]

Awesome, thanks! I'll make sure to add it to the documentation :)