Hugo: Incremental Build (while watching)

[spf13]

Hugo today rebuilds the entire site each time it is rendered. This isn't necessary and Hugo could be a lot more intelligent particularly during while watching where it already has read in the entire source and built up a tree.

What would need to happen...

1. Rethink nodes a bit and come up with a better node system, one that has a repository of nodes like pages does.
2. Add a dirty flag to pages & nodes
3. When a file change is detected mark everything that file used to touch and will touch as dirty.
4. Render & write only the dirty things.

[kindlychung]

Looking forward to this feature!

[bep]

This issue has been automatically marked as stale because it has not been commented on for at least six months.

The resources of the Hugo team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.

This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[jonathanulco]

In my view this is a huge feature request. Hugo should not restrict itself to documentation sites or small blogs but to prove to the whole of the computer services that it is today the state of the art and this even for very big websites. One of the difficulties of static site generation is the management of the generation of content in real time. Workarounds exist like making exist a hugo in page mode and hugo in site mode. I think the incremental buid could solve this problem.

[bep]

As I suspect some people will arrive here from #3962 -- and ask: "That 4 point list from @spf13 looks simple enough, why not implement that instead?"

Because this is a much harder problem. And it would be good if @spf13 could update his description to match that reality. He may be correct saying that this is a "well-understood problem" in the general sense of it, but that is like saying "so, Java and C++ have Generics, why haven't Go added support for it?"

The core of the problem comes from the freedom we give to the template developers (you): We give you a complete site object graph to play with for every template, which makes the "is this page stale?" questions harder to answer than you might think (think where clauses etc.). Taking that freedom away isn't an option, and it isn't obvious (to me at least) how to track that outside of the templates.

While still incredibly hard, this is easier and more possible now compared to when this issue was created. Back then there was no unified object graph. Now it is: The root is the Site. But it is still very difficult, and I challenge any person claiming otherwise to describe his or her solution in deeper detail and/or create a working Proof of Concept in Hugo.

[bep]

OK, so all of this rubber ducking actually helps. I think I may have wrapped my head around the hardest problem. I'll write down the gist of it so I don't forget.

In Hugo, we have done two recent improvements in the template area that is unrelated, but may help:

• We have an AST transformation of the templates to help with the total .Params.lowercase confusion.
• We have namespaced all the template funcs. They now all have a pointer receiver.

Common for all of the template executions is that they start with a template and *Page pair.

We should be able to inject a proxy in the templates so we can associate the *Page with the data it receives.

There is still a lot of work in the above, but now it at least looks possible.

/cc @budparr @spf13

[Yajo]

Why should we restrict incremental builds to watching mode?

Wouldn't it be even better if Hugo could compile static sites incrementally too?