Gatsby: Gatsby doesn't always refresh changes

[tu4mo]

Description

Gatsby sometimes fails to reload changed components.

Issue can be reproduced after version v2.24.9, so something changed between 2.24.9-2.24.10.

Steps to reproduce

https://github.com/tu4mo/gatsby-reload-issue

git clone [email protected]:tu4mo/gatsby-reload-issue.git
cd gatsby-reload-issue
npm install
npm start

In a another terminal:

node update-fakelib.js

Expected result

All changed components are reloaded.

Actual result

Not all changed components are reloaded.

Environment

  System:
    OS: macOS 11.0.1
    CPU: (8) x64 Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.15.1 - ~/.nvm/versions/node/v14.15.1/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.8 - ~/.nvm/versions/node/v14.15.1/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 87.0.4280.88
    Edge: 87.0.664.57
    Firefox: 83.0
    Safari: 14.0.1

[vladar]

Oh, I see you've already followed my advice: https://github.com/gatsbyjs/gatsby/issues/26580#issuecomment-715282412 😄

We need some way to reproduce it to move forward. Does it happen for you on any project, including our blog starter? If not, please provide a minimal reproduction and we'll be able to take another look.

[tu4mo]

Yes, I can reproduce it with blog starter. Typing something in index.js, saving quickly twice (cmd-S), making a change, saving quickly twice; doing this a few times and you'll see that the change isn't shown on the page.

[tu4mo]

[tu4mo]

I also reversed blog starter to a commit that had a version later than v2.24.0 (v2.23.12) and I can't reproduce the issue there. Clearly something changed in the refresh mechanism after v2.24.0 and made it unreliable.

@vladar Is there anything else I can do to help with this issue getting fixed? Or am I forever stuck with v2.24.0... 😞

[vladar]

I tried to reproduce it with the latest gatsby version and also on 2.24.0 but I don't see stale results. Which makes me think that it may have something to do with your environment.

Do you have Dropbox or other similar tools installed in the same folder where your gatsby site is located? Maybe some file watchers in IDE or other tools?

[vladar]

Also, can you bisect a specific version where this problem was introduced for you? The 2.23.12 - 2.24.0 range has too many changes. We need to narrow it down.

[tu4mo]

I don't have Dropbox or any other file sharing apps using code files. In my project I have watchers that built the library (Rollup and TypeScript in watch mode).

Sorry for not being clear about the version. 2.24.9 works, 2.24.10 does not.

[vladar]

Then it sounds very similar to #26192 Have you tried upgrading to 2.24.57 as suggested here: https://github.com/gatsbyjs/gatsby/issues/26192#issuecomment-690158670 ?

[tu4mo]

Tried 2.24.57, same thing, does not reload.

[tu4mo]

It feels like there's something little different in 2.24.57. After I edited a component in my library, I did get it reload by editing a Gatsby page where the component is imported. Sometimes it works, sometimes it doesn't, sometimes the whole reloading seems to freeze, and I have to restart Gatsby.

[vladar]

After I edited a component in my library, I did get it reload by editing a Gatsby page where the component is imported. Sometimes it works, sometimes it doesn't, sometimes the whole reloading seems to freeze, and I have to restart Gatsby.

But can you reproduce the problem on the blog starter with [email protected]? If not - then it means we are still stuck without any reproduction that we could use to troubleshoot your specific issue.

[tu4mo]

Yes, I can reproduce it everytime typing rapidly aaa, cmd-S, cmd-S, bbb, cmd-S, cmd-S, ccc, cmd-S, cmd-S, etc.

[tu4mo]

https://github.com/tu4mo/gatsby-reload-issue

git clone [email protected]:tu4mo/gatsby-reload-issue.git
cd gatsby-reload-issue
npm install
npm start

In a another terminal:

node update-fakelib.js

With v2.24.57 I rarely get to last update. Running it a few times and the page stops responding.

With v2.24.0 it works.

You might need to increase the amount of components "created" and decrease the sleep.

[vladar]

Thank you so much for reproduction! 💜

I definitely see how develop is getting stuck eventually. Also in the given version range - #25815 looks like a primary suspect.

Investigating this.

[vladar]

So one potential problem is here:

https://github.com/gatsbyjs/gatsby/blob/441263ed72c6ef7b0a8da581076bfa7e786c6b30/packages/gatsby/src/state-machines/develop/index.ts#L189-L195

We call markSourceFilesClean upon finishing recompiling JS. But all changes that occur during recompile are discarded. That doesn't explain why it doesn't recover on subsequent file changes when in waiting state.

One possible hint - somehow webpack stops calling invalid hook (even though chokidar actually emits change events for those):

success Building development bundle - 5.377s
[chokidar_change] fakelib\Component.js
[webpack.compiler.hooks.invalid] fakelib\Component.js
success onPreExtractQueries - 0.004s
success extract queries from components - 0.062s
success write out requires - 0.001s
[chokidar_change] fakelib\Component.js
success Re-building development bundle - 0.204s
[webpack.compiler.hooks.invalid] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js

So the next step is to dive deep into suspend/resume of the webpack. Just leaving some notes here to keep the context around.

[vladar]

OK. I think I've found the issue. This PR should fix it (hopefully): #28237

[vladar]

Published in [email protected]. Can you please check if it fixes the issue for you?

[tu4mo]

Tested, and it's better as in it doesn't crash anymore. But it still fails refreshing most of the time :(

Tried editing a Badge component in my library 4 times. Here is the browser log.

1. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 10138ms

Component didn't update.

1. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 11960ms
[HMR] Checking for updates on the server...
[HMR] Updated modules:
[HMR]  - ../lib/dist/lib/components/Badge/Badge.js
[HMR] App is up to date.

Component updated.

1. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 11997ms
[HMR] bundle rebuilding
[HMR] bundle rebuilt in 419ms

Component didn't update.

At that time I also noticed this in terminal, not sure if related:

warning Warning: Event "xstate.after(200)#waitingMachine.aggregatingFileChanges" was sent to stopped service "waitingMachine". This service has already reached its final state, and will not transition.
Event: {"type":"xstate.after(200)#waitingMachine.aggregatingFileChanges"}

1. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 532ms

Component didn't update.

Gave up after that.

[tu4mo]

I also see this in the terminal 4 times:

info changed file at /Users/tu4mo/Code/styleguide/lib/components/Badge/Badge.tsx

So Gatsby seems to pick up the change, but the rebuild times are much shorter.

[tu4mo]

I also updated Gatsby in https://github.com/tu4mo/gatsby-reload-issue and I can still reproduce this emulating multiple component updates.

[tu4mo]

@vladar Maybe you could re-open this issue?

[tu4mo]

@vladar Should I create a new issue?

[vladar]

Reopened. So to recap - it doesn't hang anymore but also doesn't emit HMR updates.

[tu4mo]

Any news?

[karlhorky]

@tu4mo you mentioned in the other thread that "Maybe the issue has something to do with symlinks..." - are you also using symlinks?

Did you ever get symlinks working with hot reloading on a previous version of Gatsby?

[tu4mo]

@karlhorky They were a suspect, but I discluded them with my reproduction example.

[karlhorky]

Ok, so you've had success with symlinks with Gatsby before? Sorry, kind of off-topic, just interested - not a lot of information on the topic.

[tu4mo]

@karlhorky Yes, I use yarn workspaces with Gatsby so some packages in node_modules are symlinks to those workspaces and they work fine. And by fine I mean they work just like any other packages, but after 2.24.9 HMR fails in general.

[karlhorky]

Cool, thanks for the heads-up. For me, I was trying to symlink mdx content into pages from a location outside of the repository, but this appears to be broken with hot module replacement / hot reloading (also requires that the dependencies like @mdx-js/mdx and such are set up in that external location).

I think I will also go for a monorepo with Yarn Workspaces.

Sorry for the slight detour in the conversation!

Thanks for bringing this issue up!

[tu4mo]

I updated the issue description with the reproduction, so it doesn't get lost in the comments.

Maybe @ascorbic has some ideas to fix this, since it might have something to do with the changes in https://github.com/gatsbyjs/gatsby/commit/9c84cb7f9d301ab6745022d600cc294b791dd9d7?