Gatsby: Changing only one file changes all files between builds

[asilgag]

This issue references Netlify and cache busting urls #11961

I commented on that issue after it was closed, so I guess nobody has reviewed it. Please consider closing this issue and reopening the previous one it that makes sense.

@KyleAMathews @jedrichards @futuregerald I am experiencing the very same when deploying to Netlify, and I have a working demo that probes that changing only one file changes ALL files between builds.

The root of the problem is webpack-runtime-[HASH].js, which changes in every build and is included in all pages.

Please repeat the following set of commands to reproduce the problem:

1) Build a gatsby site

wget https://github.com/asilgag/gatsby-benchmark/archive/master.zip
unzip master.zip
cd gatsby-benchmark-master/markdown-8000/
npm install
gatsby build

2) Save the generated "public" folder, make a change only in one file, and build it again:

mv public/ public.first
echo `date` >> src/pages/articles/2019/01/01/test-001.md 
gatsby build

3) Compare previous and current public folder, and you will see that ALL pages have changes

diff --brief -Nr public.first/ public/

Output from diff:

Files public.first/404/index.html and public/404/index.html differ
Files public.first/404.html and public/404.html differ
Files public.first/articles/2019/01/01/test-001/index.html and public/articles/2019/01/01/test-001/index.html differ
Files public.first/articles/2019/01/01/test-002/index.html and public/articles/2019/01/01/test-002/index.html differ
Files public.first/articles/2019/01/01/test-003/index.html and public/articles/2019/01/01/test-003/index.html differ
Files public.first/articles/2019/01/01/test-004/index.html and public/articles/2019/01/01/test-004/index.html differ
Files public.first/articles/2019/01/01/test-005/index.html and public/articles/2019/01/01/test-005/index.html differ
Files public.first/articles/2019/01/01/test-006/index.html and public/articles/2019/01/01/test-006/index.html differ
Files public.first/articles/2019/01/01/test-007/index.html and public/articles/2019/01/01/test-007/index.html differ
Files public.first/articles/2019/01/01/test-008/index.html and public/articles/2019/01/01/test-008/index.html differ
Files public.first/articles/2019/01/01/test-009/index.html and public/articles/2019/01/01/test-009/index.html differ
Files public.first/articles/2019/01/01/test-010/index.html and public/articles/2019/01/01/test-010/index.html differ
Files public.first/articles/2019/01/01/test-011/index.html and public/articles/2019/01/01/test-011/index.html differ
Files public.first/articles/2019/01/01/test-012/index.html and public/articles/2019/01/01/test-012/index.html differ
Files public.first/articles/2019/01/01/test-013/index.html and public/articles/2019/01/01/test-013/index.html differ
Files public.first/articles/2019/01/01/test-014/index.html and public/articles/2019/01/01/test-014/index.html differ
Files public.first/articles/2019/01/01/test-015/index.html and public/articles/2019/01/01/test-015/index.html differ
Files public.first/articles/2019/01/01/test-016/index.html and public/articles/2019/01/01/test-016/index.html differ
Files public.first/articles/2019/01/01/test-017/index.html and public/articles/2019/01/01/test-017/index.html differ
Files public.first/articles/2019/01/01/test-018/index.html and public/articles/2019/01/01/test-018/index.html differ
Files public.first/articles/2019/01/01/test-019/index.html and public/articles/2019/01/01/test-019/index.html differ
Files public.first/articles/2019/01/01/test-020/index.html and public/articles/2019/01/01/test-020/index.html differ
Files public.first/articles/2019/01/01/test-021/index.html and public/articles/2019/01/01/test-021/index.html differ
Files public.first/articles/2019/01/01/test-022/index.html and public/articles/2019/01/01/test-022/index.html differ
Files public.first/articles/2019/01/01/test-023/index.html and public/articles/2019/01/01/test-023/index.html differ
Files public.first/articles/2019/01/01/test-024/index.html and public/articles/2019/01/01/test-024/index.html differ
...

[jedrichards]

@asilgag Thanks for opening this. The thrust of my issue #11961 was about whether cache busting urls are even required on Netlify, whereas this issue relates to whether Gatsby is needlessly changing cache busting urls and so invalidating cacheable static content that references them. The two are interrelated, but having two different issues makes sense I think - and this one seems the most pertinent to Gatsby itself. Interested to see where this conversation goes 👍

[freiksenet]

Currently we don't support this. Hashes of compiled assets will change between builds, meaning all files that include them will be changed. I'll mark it as a feature request.

[asilgag]

I understand that hashes of compiled assets will change between builds, but that means that Netlify gets blocked on every build because it has to re-upload all files to its CDN.

For example, I'm building a site with ~5k pages. I change only one page, and Netlify builds the whole site in ~40-50 seconds. But then, it gets stuck deploying the whole site again to its CDN nodes.

This is a real build log from Netlify:

3:47:00 PM: Build ready to start
...
3:48:29 PM: info Done building in 50.062 sec
3:48:31 PM: Build script success
3:48:31 PM: Starting to deploy site from 'public'
3:53:40 PM: Starting post processing
3:54:03 PM: Finished processing build request in 6m59.233505763s
3:54:03 PM: Shutting down logging, 0 messages pending
4:03:44 PM: Post processing done
4:03:45 PM: Site is live

That's 50 seconds to build, and ~15 minutes update data to CDN nodes. That completely invalidates Netlify (and maybe any other similar CDN service) with medium-sized builds (1k pages and up).

In order to mitigate this problem, could it be possible to customize webpack so webpack-runtime.js file is created without any [HASH]? And then, configure Netlify to not cache that file.

Is that viable and make sense?

Or does it have some drawbacks/problems I'm not aware of?

[KyleAMathews]

Most of netlifys postprocessing is optional so you can ask them to disable some of it.

There is a way we can avoid changing the webpack hash and it's something we're looking into.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[jedrichards]

Most of netlifys postprocessing is optional so you can ask them to disable some of it.

Netlify support has manually disabled all the postprocessing they can on my ~5000 page site, but I'm still seeing long build times even when the pages are technically remaining unchanged, due to the issue detailed above. Most of the time is spent on Netlify uploading the pages to their CDN ... in my case this is taking anywhere between 5 and 10 minutes. But if the unchanged pages had stable contents between builds this would drop to near zero I think.

[federicobadini]

We are building a site with ~3k pages and we are currently suffering from the same problems.
Basically we have to wait 10 to 15 min for uploading & postprocessing.

Is there any news on this? It would really make some difference for us.

[dhmacs]

Is there any progress on this issue? Our builds for about 4k pages takes more than 20 mins

[federicobadini]

Just had a chat with Netlify support team. Our deployment time exceeds 25 minutes despite build time is only 4 to 5 minutes. From their side they are not aware of any other optimizations possible at this time.
This is quite blocking for our clients.

@KyleAMathews do you have an ETA for this? Or could we contribute somehow to investigate / speed up this?

[raisedadead]

This is a big need for us as well. Would love to be able to help in anyway we can.

At freeCodeCamp we are going to be building ~6K pages of curriculum and guide articles in 5 different languages this year. Also we would be building our Ghost powered blog ~5K pages as of this comment with Gatsby too.

Considering all this content is community powered, we would love to be able to continue using Gatsby's amazing tooling and ecosystem as we have been for a while now. Its battle tested for us and we love it.

Since we would be doing builds rather more aggressively (typically several times every day), being able to use Netlify's native hashing technology will give us that sweet fast turn around time in going live often (something we have been looking forward to the most)

While researching, I happened to stumble upon the originally closed issue and this recent comment by @brotzky, which is worth exploring for anyone landing here.

We've created a plugin that removes the cache-busting JavaScript filenames:
npmjs.com/package/gatsby-plugin-remove-fingerprints

This has improved some of our builds but 10 minutes. For example, we applied it to a site that had about 4800 generated files uploaded per build that took around 14 minutes to upload. After adding this plugin deploys were reduced to about 6 minutes.

It appears that Netlify is smart about their caching and diffing and Gatsby's cache-busting filenames were a detriment to this process.

I would also recommend adding this to gatsby-plugin-netlify as a option.

I am going to tryout the plugin myself and let you all know how it goes, thanks @brotzky for the plugin.

[KyleAMathews]

Oh hey! We forgot to close this issue as it was actually fixed recently in 2.9.0 as builds don't change as many files as before. Check out this blog post for the details. Thanks @moocar! https://www.gatsbyjs.org/blog/2019-06-12-performance-improvements-for-large-sites/#netlify-builds-faster

[hitchcott]

Has there been a regression on this?

On [email protected], i'm getting different webpack-runtime-[HASH].js files between builds.

One small change https://github.com/ethereumclassic/ethereumclassic.github.io/commit/8285ba6cc3e9efcd713e8a16295ae63a59faf888

Casues every file to get updated
https://github.com/ethereumclassic/ethereumclassic.github.io/commit/53151269e6f69bb5ee00d0f688979ac0e814c31d

The build: https://travis-ci.org/ethereumclassic/ethereumclassic.github.io/builds/621940595

Edit: This is also happening on Netlify.

[LekoArts]

Please open a new issue. Thanks!