Scully: Improve cache handling of data.json files

[BlindDespair]

🧩 Feature request

Description

Depending on server config data.json files are cached sometimes. I know we can opt out from using them in favor of index.html files, but to have the highest performance I believe data.json is the best way.
It would be great if Scully provided a mechanism which would make these files work with cache.

Describe the solution you'd like

The most straightforward thing to do in my opinion is this:

1. In puppeteerRenderPlugin add something like window['ScullyIOGenerationTime']=${Date.now()}
2. In TransferStateService in ng-lib load data.json like so:

`/data.json?gen=${ScullyIOGenerationTime}`

This will not only prevent unwanted cache, but will also work fine with being cached in between generations giving even better experience to the user.
If there are any drawbacks to this, it can be further improved I believe. But if the proposed solution is good, then I can open a PR.

Describe alternatives you've considered

Alternatively I can write a custom plugin in my project to that generation timestamp to my pages and then reprovide patched version of TransferStateService in my project. But it would be much better to solve this on Scully level.

[riscie]

We have the same request. This could also be handeled within the webserver config, but I would prefer, as @BlindDespair, to have scully handle it. An alternative to the date approach would be using the hash of the data.json file. This would have the advantage of keeping it cached, as long as it does not change.

[SanderElias]

This is a good feature request, and we will put it on our list. Don't expect a solution within a few weeks, but we will land one.

[yringler]

@SanderElias , what do you think of @BlindDespair 's solution?

In puppeteerRenderPlugin add something like window['ScullyIOGenerationTime']=${Date.now()}
In TransferStateService in ng-lib load data.json like so:
/data.json?gen=${ScullyIOGenerationTime}

Would you accept a community PR to that affect?

Traditionally, there would be some sort of hash in the file names (eg data.f30f03ddc33.json); both could be supported? I think they both have pros/cons.

having a hash/milliseconds in the JSON would make it immutable, allowing it be CDN cached to the moon.

A scully config:

jsonHash?: 'query' | 'file'

[SanderElias]

@yringler I would accept a PR for this.
The main problem is how to get the hash back into angular.
The way described has a flaw. We support partial rendering. That means, that your data.json might be generated in multiple sessions/ Scully runs.
A solution would have to cater for that. We do have incremental builds in the future, and those can not be hampered by supporting this feature.

Keeping an index would be a possibility, but if you are generating more than a few thousand pages, this might not be a good solution.
I did think about this before, but the solution isn't obvious.

[yringler]

Given that scully knows what routes a given index.html routes to, what if -

_In general_
The index.html and data.json contain maps to the hashed data file, for example

{
  '/article/1' : 'f0dd4ecf8f7fee2942a5f4d702199586'
}

Allowing the JS to know to request /article/1/data.f0dd4ecf8f7fee2942a5f4d702199586.json'

_In particular_
Scully generates 2 index.html and 2 data.json
A template file, with easily replaceable placeholders in the map
A generated file, which maps to particular data files.

For example, a template would have

{
  '/article/1' : 'ARTICLE_1_HASH'
}

And the generated file would have

{
  '/article/1' : 'f0dd4ecf8f7fee2942a5f4d702199586'
}

And (assuming incremental builds), whenever we rebuild a portion of the site, we could do a bulk update using all the template files to make sure the generated files are still referencing the latest data.json files.
Which is a lot faster then rebuilding the whole site.

So if we run a new build on /article/1, and now we have /article/1/data.4816ad212b80d20348434ba902c9685a.json
We can run a bulk replace
ARTICLE_1_HASH to 4816ad212b80d20348434ba902c9685a

The changes would have a recursive effect :( - any JSON file which references the old /article/1/data.f0dd4ecf8f7fee2942a5f4d702199586.json has to generated into a new file with a new hash, ad infinitum.

So the CDN/HTTP headers only have to be a bit looser for the index.html, and once the user has the latest index.html, he'll be browsing all the latest data at the point the HTML file was last built or generated.

I did think about this before, but the solution isn't obvious.

Truer words were never spoken 🤒