Gatsby: Hot reload doesn't re-run page query until I save twice

[aaronadamsCA]

When I modify a page query, I have to save twice for hot reload to re-run the page query.

Steps to reproduce

1. Edit a page query
2. Save
3. Save

Expected result

run page queries output after first save, for example:

success onPreExtractQueries - 0.009s
success extract queries from components - 0.032s
success write out requires - 0.002s
success run page queries - 0.011s - 1/1 88.96/s
success Re-building development bundle - 0.064s

Actual result

run page queries output only after second save:

success onPreExtractQueries - 0.009s
success extract queries from components - 0.045s
success write out requires - 0.006s
success Re-building development bundle - 0.425s
success onPreExtractQueries - 0.009s
success extract queries from components - 0.032s
success write out requires - 0.002s
success run page queries - 0.011s - 1/1 88.96/s
success Re-building development bundle - 0.064s

I don't think this is a recent thing; I just didn't entirely recognize the issue I was having with hot reloads until now. "Try saving twice" didn't really occur to me until this week, when I discovered the workaround.

Environment

Note my environments are WSL 2 on Windows 10. Easily reproduced in this repo; just edit the page query in src/pages/index.js and note you'll need to save twice to get the page query to re-run.

  System:
    OS: Linux 4.19 Debian GNU/Linux 9 (stretch) 9 (stretch)
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Shell: 4.4.12 - /bin/bash
  Binaries:
    Node: 12.18.3 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/bin/yarn
    npm: 6.14.6 - /usr/local/bin/npm
  Languages:
    Python: 2.7.13 - /usr/bin/python
  npmPackages:
    gatsby: ^2.24.47 => 2.24.47
    gatsby-source-sanity: ^6.0.4 => 6.0.4

and

  System:
    OS: Linux 4.19 Ubuntu 18.04.1 LTS (Bionic Beaver)
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Shell: 4.4.19 - /bin/bash
  Binaries:
    Node: 11.6.0 - /usr/bin/node
    npm: 6.14.8 - /usr/bin/npm
  Languages:
    Python: 2.7.15 - /usr/bin/python
  npmPackages:
    gatsby: ^2.24.47 => 2.24.47
    gatsby-source-sanity: ^6.0.3 => 6.0.3
  npmGlobalPackages:
    gatsby-cli: 2.4.11

[codepunkt]

Same problem here, also running on WSL 2.
This works with older gatsby versions, the double refresh thing is new. I think i was running a 2.22.x when it still worked.

[aaronadamsCA]

I can confirm it works correctly in WSL 2 on 2.22.22. That's a great detail to add, thanks @codepunkt.

[mrseanbaines]

Also getting this even when I change styles (styled-components) - sometimes have to save again before changes show

[JimmyBeldone]

Same problem here, rolling back to 2.24.9 version seems to work according to this issue #26192

[tu4mo]

Also have a similar problem. Changes are not reloading. Sometimes saving twice helps, sometimes not. Works fine on 2.24.9. I'm on a Mac, so not related to WSL.

[tu4mo]

I should probably add that I'm using GATSBY_HOT_LOADER=fast-refresh.

[icy-meteor]

Can confirm, facing this issue.

[AnthonyLedru]

I'm also facing the issue on Mac OS.
The only workarround that worked for me is to rollback to 2.24.9

[hartshorne]

I've had this issue intermittently as well. Looking at the changes between 2.24.9 and 2.24.10:

https://github.com/gatsbyjs/gatsby/compare/[email protected]@2.24.10

It looks like @ascorbic did some work in https://github.com/gatsbyjs/gatsby/pull/25815 that might have affected how fast-refresh and/or hot-reload work. @ascorbic what do you think?

[hartshorne]

I am using macOS 10.15.7, GATSBY_HOT_LOADER=fast-refresh, and the project uses TypeScript 4.0.3. I'm also using the --https flag. Feels like some kind of timing issue, so it may be hard to build a minimally-reproducible example. In our medium-sized project, this is what I'm experiencing:

after editing and saving src/colors.ts:

Browser:

[HMR] connected
[HMR] bundle rebuilding
[HMR] bundle rebuilt in 13871ms
[HMR] Checking for updates on the server...
[HMR] bundle rebuilding
Ignored an update to unaccepted module ./src/colors.ts -> ./src/templates/topExtension.tsx -> ./.cache/_this_is_virtual_fs_path_/$virtual/sync-requires.js -> ./.cache/app.js -> 1
[HMR] The following modules couldn't be hot updated: (Full reload needed)
This is usually because the modules which have changed (and their parents) do not know how to hot reload themselves. See https://webpack.js.org/concepts/hot-module-replacement/ for more details.
[HMR]  - ./src/colors.ts
[HMR] Reloading page

Gatsby:

You can now view [project] in the browser.
⠀
  Local:            https://localhost:8000/
  On Your Network:  https://[ip]:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
  Local:            https://localhost:8000/___graphql
  On Your Network:  https://[ip]:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use gatsby build
⠀
success Building development bundle - 20.171s
success onPreExtractQueries - 0.005s
success extract queries from components - 0.187s
success write out requires - 0.009s
success Re-building development bundle - 13.893s

At this point, the server is frozen:

% curl --verbose https://localhost:8000/                                                                                                                                           (git)-[main] 
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=localhost
*  start date: Sep  9 18:02:31 2020 GMT
*  expire date: Dec 13 18:02:31 2022 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=devcert
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.64.1
> Accept: */*
>

If you % touch src/colors.ts, then the browser reloads the page, the change is visible, and the browser console is blown away (from the reload) and says [HMR] connected.

Gatsby:

success onPreExtractQueries - 0.006s
success extract queries from components - 0.202s
success write out requires - 0.009s
success Re-building development bundle - 0.616s

And the server works again:

% curl --verbose https://localhost:8000/                                                                                                 
[...]
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Last-Modified: Wed, 07 Oct 2020 21:10:43 GMT
< ETag: W/"23c-17504e6ec1c"
< Content-Type: text/html; charset=UTF-8
< Content-Length: 572
< Date: Wed, 07 Oct 2020 22:57:21 GMT
< Connection: close
< 
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
<!DOCTYPE html><html><head><meta charSet="utf-8"/><meta http-equiv="x-ua-compatible" content="ie=edge"/><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/><meta name="note" content="environment=development"/><title data-react-helmet="true"></title><link rel="sitemap" type="application/xml" href="/sitemap.xml"/><style data-linaria-critical="/"></style><script src="/socket.io/socket.io.js"></script></head><body><div id="___gatsby"></div><script src="/polyfill.js" nomodule=""></script><script src="/commons.js"></script></body></html>%

[ascorbic]

@hartshorne It does sound like it's related to that, but as you said it's hard to reproduce. I've not managed to reproduce it myself, which makes it a challenge to fix. Can you run gatsby develop with the --verbose flag and share the logs?

[ascorbic]

Can you also confirm whether it's reproducible without https

[hartshorne]

Sure, running:

GATSBY_HOT_LOADER=fast-refresh ./node_modules/.bin/gatsby develop --host=0.0.0.0 --https --verbose

Outputs:

verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "develop"
verbose loading local command from:
verbose running command: develop
info setting up automatic SSL certificate (may require elevated permissions/sudo)

verbose Transition to "initializing"
success open and validate gatsby-configs - 0.076s
success load plugins - 0.126s
success onPreInit - 0.026s
success initialize cache - 0.004s
success copy gatsby files - 0.037s
success onPreBootstrap - 0.011s
verbose Transition to "initializingData"
success createSchemaCustomization - 0.007s
verbose Transition to "initializingData" > "sourcingNodes"
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.007s
success source and transform nodes - 0.142s
verbose Transition to "initializingData" > "buildingSchema"
success building schema - 0.281s
verbose Transition to "initializingData" > "creatingPages"
info Total nodes: 1075, SitePage nodes: 485 (use --verbose for breakdown)
verbose Number of node types: 9. Nodes per type: SitePlugin: 31, Site: 1, SiteBuildMetadata: 1, Directory: 9,
success createPages - 0.669s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 484 changed pages
success Checking for changed pages - 0.003s
verbose Transition to "initializingData" > "creatingPagesStatefully"
success createPagesStatefully - 0.440s
verbose Transition to "initializingData" > "rebuildingSchemaWithSitePage"
success update schema - 0.036s
verbose Transition to "initializingData" > "writingOutRedirects"
success write out redirect data - 0.003s
verbose Transition to "initializingData" > "done"
verbose Transition to "runningPostBootstrap"
success onPostBootstrap - 0.004s
info bootstrap finished - 3.394s
verbose Transition to "runningQueries"
success onPreExtractQueries - 0.003s
success extract queries from components - 1.153s
verbose Transition to "runningQueries" > "writingRequires"
success write out requires - 0.046s
verbose Transition to "runningQueries" > "calculatingDirtyQueries"
verbose Transition to "runningQueries" > "runningStaticQueries"
verbose Transition to "runningQueries" > "runningPageQueries"
success run page queries - 0.119s - 65/65 546.87/s
verbose Transition to "runningQueries" > "waitingForJobs"
verbose Transition to "runningQueries" > "done"
verbose Transition to "startingDevServers"
⠀
You can now view ids in the browser.
⠀
  Local:            https://localhost:8000/
  On Your Network:  https://192.168.170.192:8000/
⠀
View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
  Local:            https://localhost:8000/___graphql
  On Your Network:  https://192.168.170.192:8000/___graphql
⠀
Note that the development build is not optimized.
To create a production build, use gatsby build
⠀
success Building development bundle - 20.771s
verbose Transition to "waiting"

At this point, the dev server works. If I edit src/colors.ts, for example, this is output next:

verbose Transition to "waiting" > "aggregatingFileChanges"
verbose Webpack file changed: /Users/beau/browser/src/colors.ts
verbose Transition to "runningQueries"
success onPreExtractQueries - 0.008s
success extract queries from components - 0.195s
verbose Transition to "runningQueries" > "writingRequires"
success write out requires - 0.008s
verbose Transition to "runningQueries" > "calculatingDirtyQueries"
verbose Transition to "runningQueries" > "runningStaticQueries"
verbose Transition to "runningQueries" > "runningPageQueries"
verbose Transition to "runningQueries" > "waitingForJobs"
verbose Transition to "runningQueries" > "done"
verbose Transition to "recompiling"
success Re-building development bundle - 13.868s
verbose Webpack file changed:
verbose Transition to "waiting"

At this point, the development server is not responding; curl https://localhost:8000/ just hangs. But if I touch src/colors.ts, I see this:

verbose Transition to "waiting" > "aggregatingFileChanges"
verbose Transition to "runningQueries"
success onPreExtractQueries - 0.008s
success extract queries from components - 0.168s
verbose Transition to "runningQueries" > "writingRequires"
success write out requires - 0.008s
verbose Transition to "runningQueries" > "calculatingDirtyQueries"
verbose Transition to "runningQueries" > "runningStaticQueries"
verbose Transition to "runningQueries" > "runningPageQueries"
verbose Transition to "runningQueries" > "waitingForJobs"
verbose Transition to "runningQueries" > "done"
verbose Transition to "recompiling"
success Re-building development bundle - 0.500s
verbose Transition to "waiting"

The server responds, and everything works as expected.

[hartshorne]

@ascorbic enabling/disabling --https does not change the output or behavior, same problem with or without.

[ascorbic]

Thanks. That's interesting, because from the log it seems that everything is happening as expected. The fact that the server is frozen is an interesting one though, as we don't pause it or anything like that. It seems to be a specific issue with the Webpack Dev server. The penultimate line where it says that a file has changed but there is no file listed is an interesting one though.

[aaronadamsCA]

I also tried with --verbose, the only difference between save 1 and save 2 is the appearance of an additional line:

  verbose Transition to "runningQueries" > "runningPageQueries"
+ success run page queries - 0.026s - 1/1 37.81/s
  verbose Transition to "runningQueries" > "waitingForJobs"

If it helps, here's a "stress test" so to speak which might change your thoughts on what's happening:

1. Add a letter to the name of the GraphQL page query
2. Save
3. GOTO 1

I am seeing page queries re-run on _every other_ save to the file.

  query IndexPagee {

success onPreExtractQueries - 0.013s
success extract queries from components - 0.090s
success write out requires - 0.004s
success Re-building development bundle - 0.357s

  query IndexPageee {

success onPreExtractQueries - 0.016s
success extract queries from components - 0.092s
success write out requires - 0.004s
success run page queries - 0.022s - 1/1 45.91/s
success Re-building development bundle - 0.448s

  query IndexPageeee {

success onPreExtractQueries - 0.010s
success extract queries from components - 0.064s
success write out requires - 0.004s
success Re-building development bundle - 0.351s

  query IndexPageeeee {

success onPreExtractQueries - 0.006s
success extract queries from components - 0.046s
success write out requires - 0.007s
success run page queries - 0.029s - 1/1 34.59/s
success Re-building development bundle - 0.318s

[hartshorne]

@ascorbic I checked out a few starter projects, fast-refresh seems to work fine there. does fast-refresh work on the main gatsbyjs.com site? (i started building https://github.com/gatsbyjs/gatsby/tree/master/www, but stopped when it started resizing some thousands of images.)

@aaronadamsCA my first touch (or save) breaks/blocks the server, second one unbreaks it. do you see the same behavior with curl http://localhost:8000 -- is the server frozen for you too?

also @aaronadamsCA (or others experiencing this issue) -- here are my dependencies, curious what overlap we have:

"dependencies": {
  "@babel/core": "^7.11.6",
  "@react-spring/parallax": "^9.0.0-rc.3",
  "@stripe/stripe-js": "^1.9.0",
  "core-js": "^3.6.5",
  "firebase": "^7.22.1",
  "gatsby": "2.24.72",
  "linaria": "^1.3.3",
  "nested-error-stacks": "^2.1.0",
  "prism-react-renderer": "^1.1.1",
  "react": "^16.13.1",
  "react-dom": "^16.13.1",
  "react-fast-compare": "^3.2.0",
  "react-google-recaptcha": "^2.1.0",
  "react-helmet": "^6.1.0",
  "react-redux": "^7.2.1",
  "react-use-measure": "^2.0.2",
  "redux": "^4.0.5",
  "reselect": "^4.0.0",
  "rollbar": "^2.19.3",
  "scheduler": "^0.19.1",
  "uuid": "^8.3.1",
  "window.requestanimationframe": "^1.0.1",
  "woopra": "^1.0.2"
},
"devDependencies": {
  "@types/google.analytics": "^0.0.40",
  "@types/nested-error-stacks": "^2.1.0",
  "@types/react": "^16.9.51",
  "@types/react-dom": "^16.9.8",
  "@types/react-google-recaptcha": "^2.1.0",
  "@types/react-helmet": "^6.1.0",
  "@types/react-redux": "^7.1.9",
  "@types/scheduler": "^0.16.1",
  "@types/uuid": "^8.3.0",
  "@typescript-eslint/eslint-plugin": "^4.4.0",
  "@typescript-eslint/parser": "^4.4.0",
  "babel-preset-gatsby": "^0.5.13",
  "eslint": "^7.10.0",
  "eslint-config-prettier": "^6.12.0",
  "eslint-plugin-import": "^2.22.1",
  "eslint-plugin-prettier": "^3.1.4",
  "eslint-plugin-react": "^7.21.3",
  "eslint-plugin-react-hooks": "^4.1.2",
  "eslint-plugin-simple-import-sort": "^5.0.3",
  "gatsby-cli": "^2.12.106",
  "gatsby-plugin-linaria": "^2.1.0",
  "gatsby-plugin-react-helmet": "^3.3.13",
  "gatsby-plugin-sitemap": "^2.4.15",
  "gatsby-plugin-typescript": "^2.4.21",
  "gatsby-plugin-why-did-you-render-redux": "^0.2.1",
  "gatsby-source-filesystem": "^2.3.34",
  "gatsby-transformer-javascript-frontmatter": "^2.3.14",
  "gatsby-transformer-json": "^2.4.14",
  "husky": "^4.3.0",
  "lint-staged": "^10.4.0",
  "prettier": "^2.1.2",
  "redux-devtools-extension": "^2.13.8",
  "rollbar-sourcemap-webpack-plugin": "^2.12.0",
  "sort-package-json": "^1.46.0",
  "stylelint": "^13.7.2",
  "stylelint-config-rational-order": "^0.1.2",
  "stylelint-config-recommended": "^3.0.0",
  "stylelint-order": "^4.1.0",
  "stylelint-prettier": "^1.1.2",
  "ts-node": "^9.0.0",
  "typescript": "^4.0.3"
}

[aaronadamsCA]

@aaronadamsCA my first touch (or save) breaks/blocks the server, second one unbreaks it. do you see the same behavior with curl http://localhost:8000 -- is the server frozen for you too?

@hartshorne, nope! My server continues running. My project isn't using TypeScript though, I'm able to reproduce with a very small set of deps:

https://github.com/shopnishe/gatsby-issues/blob/0449fa53c63b76409ce26190c0e2212c39f9f997/package.json#L16-L29

[vladar]

I stumbled upon this issue while debugging a somewhat similar scenario. And now I think it is exactly the same problem. If so, it should be fixed by #27408

[vladar]

Published in [email protected]

[hartshorne]

@vladar thank you for digging into this. I tried upgrading, gatsby clean, and still run into this issue. I tried increasing PAGE_QUERY_ENQUEUING_TIMEOUT to 1000, but that did not seem to help. Anything I can do to track this down on my end?

[tu4mo]

Still happening for me also. And now everything also freezes after I change some files.

[vladar]

@aaronadamsCA can you please check if it fixes your original issue?

If it does, then @hartshorne probably hits something else which warrants a separate issue. If not, then the problem I was chasing is probably unrelated to this one and I will re-open it.

[vladar]

Also, to clarify. There seem to be two different issues (at least):

1. When editing the query (without editing the code) - we were not running this query, so hot reload was providing old results (page refresh didn't help here)

2. When editing both the query and the code - we don't synchronize when the recompiled bundle and data changes are pushed. So you may get the new data but old javascript or the new javascript but old data, which can cause runtime errors (but page refresh helps here)

I've only fixed the first scenario. Synchronization should be tackled separately as the source of the problem is very different. Maybe there are others as well. All deserve their own bug reports as they are likely different problems.

[hartshorne]

@aaronadamsCA does @vladar's change resolve your issue? It does not fix mine or @tu4mo's issues from the looks of it.

[aaronadamsCA]

@vladar, I can confirm this change has resolved my issue, thank you! Changes to page and template queries now re-run immediately and the changes hot reload. 👍 (For what it's worth hot reload appears to be better in general now; I'm also noticing my gatsby-source-sanity watch mode is suddenly much more reliable.)

@hartshorne, my guess is you'll want to file a new issue, since I think your problem (may have been exacerbated by this problem but) is somewhere in the Gatsby TypeScript stack. I will soon be incrementally adopting TS in Gatsby so for my sake I hope you find the fix 😅

[tu4mo]

@vladar Yep, I think this solved the random issue, but I still have another issue with some components not updating at all. Refreshing doesn't do anything, only restarting. The components come from a library that's build in another yarn workspace. Maybe the issue has something to do with symlinks...

[AnthonyLedru]

The changes made by @vladar also fixed my issue. 👍

[vladar]

@tu4mo I've definitely seen issues with file watching in complex cases, e.g. when using dropbox and similar tools. If you can reliably reproduce it, feel free to open a new issue! Thanks.

[hartshorne]

@vladar there seems to be some kind of race with cache files that https://github.com/callstack/linaria produces. Here's a new issue: https://github.com/gatsbyjs/gatsby/issues/27626 and a minimal repro: https://github.com/hartshorne/hot-reload-repro

[hartshorne]

@tu4mo are you using https://github.com/callstack/linaria?

[tu4mo]

@hartshorne No. Some kind of cache race condition sounds possible though. It takes a second or two to build my component library on save. Gatsby seems to detect the change looking at the cli and browser console. But no changes show in the browser.

[tujoworker]

I experience the same, 100% like you @tu4mo describe. The code is in this branch. I not use Linaria (but Emotion). But I use mdx. Both, changes in JS or MDX files causes the "freeze" randomly. Most if I "safe" often, or quickly one after the other.

But I also experience, that the safe & refresh gets slower after a while.

[tu4mo]

@tujoworker Please open a new issue with the information you have so they can fix this.