Gatsby: Why is Gatsby using watchers for `gatsby build`

[jessepinho]

Summary

Relevant information

When running gatsby build, my CI system (Jenkins) often hits the system limit for file watchers. The limit on the system is 1024. I have only about 700 files in the repo, but this issue seems to occur when multiple builds are running simultaneously.

But my question is, why would gatsby build require file watchers? It seems that that would only be needed for watch commands. Is this a mistake?

Environment (if relevant)

  System:
    OS: macOS 10.14.5
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.5.1 - node_modules/.bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Languages:
    Python: 2.7.16 - /usr/local/bin/python
  Browsers:
    Chrome: 75.0.3770.142
    Firefox: 66.0.2
    Safari: 12.1.1
  npmPackages:
    gatsby: ^2.13.14 => 2.13.14 
    gatsby-image: ^2.0.30 => 2.0.37 
    gatsby-plugin-react-helmet: ^3.0.7 => 3.0.12 
    gatsby-plugin-styled-components: ^3.0.6 => 3.0.7 
    gatsby-plugin-typescript: ^2.1.1 => 2.1.1 
    gatsby-transformer-remark: ^2.3.0 => 2.3.8 

File contents (if changed)

Hm, these are all private as it's a private repo. Are these really needed?

gatsby-config.js: N/A
package.json: N/A
gatsby-node.js: N/A
gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[pieh]

Because used package for globbing + watching (chokidar) doesn't have way to disable watching. We were looking for alternatives, but there were none that actually allow to disable watching. We could use different packages for builds (so only globbing, without watching) and develop (globbing + watching) - but this introduce potential to cause differences between builds and develop

[jessepinho]

Ah OK, makes sense. I should open an issue with chokidar then? I'll close this one in the meantime.

[antoinerousseau]

We could use different packages for builds (so only globbing, without watching) and develop (globbing + watching) - but this introduce potential to cause differences between builds and develop

There are differences between builds and develop anyway. I think it would be useful for issues like #11406 and others to use a different library for each case.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[lucascaro]

Hey guys, I'm working on a project and seeing build failures due to this issue fairly often. Are there any plans to update gatsby to not use chokidar during build? If needed for globbing, maybe using the same library that chokidar uses for globbing?

[jessepinho]

Or, would the maintainers be open to a PR that replaces chokidar with
something else for production builds?

On Fri 15. Nov 2019 at 00:05 Lucas Caro [e-mail] wrote:

Hey guys, I'm working on a project and seeing build failures due to this
issue fairly often. Are there any plans to update gatsby to not use
chokidar during build? If needed for globbing, maybe using the same library
that chokidar uses for globbing?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/15843?email_source=notifications&email_token=AAIR2CC7WUZJI7NUPBOALWDQTXKULA5CNFSM4IESELY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEDUOHI#issuecomment-554125085,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIR2CAVE4A5WHWVGA25LZLQTXKULANCNFSM4IESELYQ
.

[wardpeet]

Depends on what the package is and how you're going to solve it. Fs is brittle so it's not always easy to test all edge cases.

[jessepinho]

Hm, ok. Are there particular edge cases in mind (i.e., do they have tests)?
Just wondering what I’d be getting myself into 😆

On Fri 15. Nov 2019 at 17:23 Ward Peeters [e-mail] wrote:

Depends on what the package is and how you're going to solve it. Fs is
brittle so it's not always easy to test all edge cases.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/gatsbyjs/gatsby/issues/15843?email_source=notifications&email_token=AAIR2CDY2SOQFYAAVILEH2DQT3EJ5A5CNFSM4IESELY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEF6PPI#issuecomment-554428349,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIR2CAUHI6T55ROEFNA6V3QT3EJ5ANCNFSM4IESELYQ
.

[sidharthachatterjee]

@jessepinho Great question! One I can think of right off the top of my head is hat choikdar seems to ignore changes when running in some environments (CI being the one in question). This made us skip some hot reloading tests on CircleCI 😒

I _did_ attempt to move over to a different file watcher (watchpack) in https://github.com/gatsbyjs/gatsby/pull/15854 (it fixed the CI issue)

Feel free to take that over if you like?

In the mean time, let's close this issue? We should move the discussion (of moving over to a different watcher in a separate issue in my opinion, feel free to open that!)

[esturcke]

@jessepinho I was running into this issues as well in our restrictive Jenkins environment and settingCHOKIDAR_USEPOLLING=1 got around the file watcher limit problem.