Graphql-code-generator: High CPU usage in watch mode due to polling (chokidar)

[slikts]

The watch mode uses chokidar with polling, which in a large project means high CPU usage; in my case (thousands of source files) it uses almost an entire core at idle. Polling is useful in some compatibility scenarios, but it shouldn't be the default, and it should be configurable. I've currently
worked around the issue by making my own watch mode with some additional enhancements like triggering the build selectively (only for source files that actually feature GraphQL), and I can confirm that chokidar works fine without polling.

[dotansimha]

Hi @slikts , thank you for reporting it. I can guess that the solution for the selective watch it the one you suggested in https://github.com/dotansimha/graphql-code-generator/issues/1797 .

[slikts]

You're welcome; I wish I could share my implementation, but it's rudimentary and somewhat project specific. I've also incorporated my workaround for avoiding unnecessary builds from #1570, i.e., the watch script updates the cache so that running the build script doesn't rebuild unnecessarily.

[s-h-a-d-o-w]

Why is polling used in the first place? Based on chokidar's docs, it has non-polling solutions for all platforms?
I'm on Windows, set usePolling to false and while it used to max out all of my cores, it now sits at 0% CPU usage. (And it of course still works... ;) )

I should also note that I'm asking because I considered submitting a PR but then thought "Well but despite what the docs say, maybe they experienced issues on macOS and/or Linux."

[paales]

We're running into the High CPU usage as well. Especially on lower end systems (not every developer has the newest laptop) it becomes an issue. Would be nice if we could switch to file system events :)

[paales]

Apparently I'm retarded and need to read the documentation, the following solved it for me:

documents:
  - '**/*.graphql'
  - '!generated/*.graphql'
  - '!node_modules/**'

[mikelovesrobots]

I was also seeing a node process pegged at 100% cpu for hours at a time and can verify that it's because chokidar was told to use polling rather than the default and lightweight filesystem events. https://github.com/dotansimha/graphql-code-generator/blob/80a1e6726c42e5a74d2717befb0834d94b515df4/packages/graphql-codegen-cli/src/utils/watcher.ts#L97

Looking at the docs for usePolling. I think this should be set to false or deleted entirely and let chokidar make the call, but I'm not the maintainer, and maybe some users were complaining cause --watch wasn't working on the network drives they were doing development on or something.

In the meantime, as a user you can avoid this whole garbage fire by setting the environment variable CHOKIDAR_USEPOLLING=0 which will tell chokidar to ignore anyone telling it to use polling. Hooray, no more 100% cpu process on osx when the machine is completely idle.

[paales]

@mikelovesrobots That is giving me issues like:

(node:61052) UnhandledPromiseRejectionWarning: Error: EMFILE: too many open files, watch
    at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:127:28)

It seems to be watching all files in the node_modules folder as well, which isn't really a thing when using polling, but to use file system events chokidar probably needs to be configured differently?

[mikelovesrobots]

@paales Oh that's wild. We've also got a documents section in our config file reducing the watched scope to just the things that we think matter:

overwrite: true
generates:
  ./types/landing-gatsby-graphql-generated.ts:
    schema: https://localhost:8002/___graphql
    documents:
      - landing/src/**/*.{ts,tsx}
      - ./node_modules/gatsby-*/**/*.js
    config:
      gqlImport: gatsby#graphql
      namingConvention:
        enumValues: keep
    plugins:
      - typescript
      - typescript-operations

Sounds like turning off usePolling wasn't the universal upgrade I thought it was.

[paales]

@mikelovesrobots I've reduced the scope of the files that it scans and doesn't run into issues anymore and your proposed solution does work.