Gatsby: Large sites ~200k pages, incremental builds?

[Tawfiqh]

Summary

We have a large site of about 200k pages and we don't want to have to rebuild the whole site every time we make a change and deploy.

Relevant information

A lot of our pages are set pages that pull in a lot of data and take a while to build but they won't change often.

We could just build them in a separate Gatsby project to the rest of our site. Having two Gatsby projects set-pages and main-site and then combining the two built /public directories. But as far as I can tell, that won't allow us to use the <Link to=...> component to get quick SPA performance
when transitioning between pages from the main-site and the set pages.

Ideal situation

Ideally we want to build those "set-pages" once, and then each time we update our gatbsy project we don't have to rebuild them.

Is there a good/recommended way we can selectively rebuild just parts of the site on deploy? We would prefer not to rebuild the whole site on every deployment.

Current attempts

Currently our best thinking is to split the set-pages into a separate project from the main-site, combine the two outputs and just accept that when linking between the two we should use an <a href...> instead of a <Link to=...>.

Have read a lot of closed issues on this I can see that the faster build times in Gatsby v2 help alleviate this issue for a lot of sites, but 200k pages is where we are and we're wondering if build times will be manageable as we increase scale. Mainly we don't want small blog updates having to rebuild the entire set-pages.

[pieh]

At this moment it's not possible to do incremental builds. It's definitely something we will want to support but it's not trivial to do as currently all code expects that everything is built when running gatsby build.

Here's open issue for that https://github.com/gatsbyjs/gatsby/issues/5002, so I'll close this to not have multiple open duplicate issues

[dcworldwide]

I also have a site with a large # of pages (160k and growing). I need partial updates (1k page updates a day). Would love to learn more about whether this can be solved.

[justinwilson-apollidon]

+1 for this.

[dskvr]

+1

[parkerproject]

+1

[cullophid]

+1

[DSchau]

@fenchi thanks for showing your support, but if you can just react to the original issue with an emoji, that's a much more helpful way of showing your support. Thanks!

(For full context, we can then sort issues by # of reactions, which can be a helpful signal to help us prioritize)

[sanjaySP]

+1

[b1lb0]

+10k :)