Eleventy: --incremental Build, only process files that have changed.

[zachleat]

This is the step that requires knowledge of each template’s dependencies:

• What collections the changed template is a part of and what other templates _consume_ those collections.
• Keep in mind JavaScript dependencies (for 11ty.js)
• If the file is a _layout_, rebuild other templates use the layout.
• If the file is a data file, rebuild what templates use the data (global data will require some extra work but directory and template data are straightforward)
  
  
  
  • Stretch goal here for _huge_ pagination templates is that it would be super nice to know what changed in the data file to have a bit more precision about what pagination templates we need to write. (Selfishly I want this for my twitter archive project) (Filed separately at #1087)
  
  
• Passthrough copy files filed separately as #977

Punt into separate issues, later:

• If the file is a config file, package.json, rebuild everything for now.
• If the file is an include or an extend or an import or other internal-to-template-language specific feature, how do we discover what other files consume this? Maybe no way to tell for relative includes that don’t live in the includes folder?

[zachleat]

The really difficult part about this task is that doing it correctly would require a detailed dependency map of the content, specifically around what uses what collections.

[insanetisane]

I believe the pug cli watch option does this. Even if included css file changes it only rebuilds templates that reference it.
It’s very smart. Not sure how it’s done tho.

[zachleat]

This repository is now using lodash style issue management for enhancements (see https://twitter.com/samselikoff/status/991395669016436736)

This means enhancement issues will now be closed instead of leaving them open. The enhancement backlog can be found here: https://github.com/11ty/eleventy/issues?utf8=%E2%9C%93&q=label%3Aneeds-votes+sort%3Areactions-%2B1-desc+

[zachleat]

Use Object.defineProperties to add getters to track dependencies between templates and data.

[zachleat]

Proxies and Sorted Directed Acyclic Graphs, y’all.

https://www.npmjs.com/package/toposort
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy

[zellwk]

Just adding another thought: I wonder how would incremental watch mode work with a task runner like Gulp and browser sync?

[jakearchibald]

If/when this happens, it'd be great if the optimisations were available outside of watch mode.

TypeScript does this via .tsbuildinfo files.

This means that users that are using different watchers would benefit from the incremental build too.

[Ryuno-Ki]

Hm, some people referred different existing implementations. Could we gather their source code here so to lay out a plan?
(If we get lucky, we find shared libraries among them!)

[zachleat]

Did a really dumb/rough start of this here https://github.com/11ty/eleventy/commit/6e294bd7160d44b400810bab84273336009c99db

[zachleat]

After working on https://www.zachleat.com/web/own-my-tweets/ I’m also thinking a nice feature might be some smarts about pagination data too, specifically adding entries into a huge paginated array would only write files for the new/changed entries.

[zachleat]

@matthewp had an interesting idea about reading files in the output directory to compare contents to avoid re-writing and save on rsync tasks https://twitter.com/matthewcp/status/1174013502215852032

[zachleat]

I did a fair bit of work to experiment playing around with @matthewp’s idea to compare output before writing is in the incremental-file-hash-cache branch but it wasn’t quite good enough. Will require more work but might get scrapped. https://github.com/11ty/eleventy/tree/incremental-file-hash-cache

Hashing the content seemed to be fast enough (using farmhash https://www.npmjs.com/package/farmhash) but writing an extra file for the cache seemed to negate the gains here.

Some super quick numbers (farmhash versus node’s built-in crypto module)

Yeah, not writing any files did provide a small performance improvement for large projects (but at the cost of up-front hashing for the first run to create the cache file)

I dunno, just not quite sold here on this yet.

[matthewp]

@zachleat Thanks for giving it a try! For clarity, I wasn't expecting the comparison to speed up builds, I was expecting it to speed up rsync.

Out of curiosity, was this hashing implementation a separate hash file for every page, or did you concat these some how into 1 hashed file so you only need to do 1 read.

From an implementation simplicity perspective you could skip the hashing and just read in the current version of the file. This would indeed be slower, but how much slower?

[piperhaywood]

Another thought about this:
If --incremental or something similar is too difficult, a different approach that might help is to exclude files via the config file instead of .eleventyignore. This way, we can use the preexisting --config flag to swap configs when developing locally.

My specific use case:
We’re pulling in data from an API on each build. Right now there are only 20 pages on the whole site, but the build time is taking anywhere from 9 to 50 seconds each time due to the API. Ideally we wouldn’t hit that API every time, it’s only really necessary when the site is deployed to Netlify. If I could exclude that particular file within an Eleventy config file such as eleventy.dev.js, then I could run npx @11ty/eleventy --serve --config=eleventy.dev.js for much faster development.

[dceddia]

This feature would be awesome to have. I was thinking about how to do the dependency mapping and how it seems like each template language would need to be able to do its own map.

Poking around Nunjucks docs I found the load event (here). It sounds potentially useful! They say:

The 'load' event gets emitted whenever a Loader retrieves the source of a template. It can be listened to in order to determine template dependencies at runtime.

Still gotta build up that DAG I guess, but this might be a step in the right direction, for Nunjucks anyway.

[bep]

I wasn't expecting the comparison to speed up builds, I was expecting it to speed up rsync.

rsync has a --checksum flag.

On an added note, I'm glad to see that I'm not the only having an hard time wrapping my head around "partial builds" :-)

[zachleat]

I’m dividing this work into smaller issues and orchestrating on a GitHub project here: https://github.com/orgs/11ty/projects/3

[Lohn]

Just wondering... And If we use some command to to build a specific file? Not the builder find this file, but we explicitly pointing it.
Something like:

eleventy --build-incremental _blog/2020/some_nice_title_here.md

I asking about this, because when you update one post/note/photo, you (our your CMS) know what was changed. Maybe is easier (and less resourceful) this way.

[Ryuno-Ki]

I doubt, that this would result in a nice experience.
Imagine, that you have a year index page for all blog pages published in 2020.
This one wouldn't be updated, which could lead in confusion.

In technical terms: There's a dependency tree, which would flag other sites as „dirty”, i.e. in need of recomputation.

[Lohn]

I understand. But anyway the incremental build won't identify only the file changed?
So, it needs to rebuild the index page too. I was wondering about a way to avoid the "hash search" of all the files to make the incremental build less time/resource consuming.