gatsby-source-filesystem fails with "EMFILE: too many open files" on Windows with lots of files

[FraserThompson]

Description

When I try to gatsby build a site with around 9,500 files in the src folder on Windows 10 I get this error and the build fails:

Error: EMFILE: too many open files

It specifies the error occurred on an image file (usually a different one each time).

Steps to reproduce

My project is an incredibly image heavy website, and it uses gatsby-source-filesystem, gatsby-image and gatsby-transformer-sharp to process the images.

I have lots of folders containing lots of images like this:

content/gigs/[gig name]/[artist_name]/[image1.jpg,image2.jpg,etc]

In each gig folder there's an index.md which contains metadata for the page. I use a graphql query in gatsby-node.js to turn these markdown files into pages via a template. The template uses a graphql querie and a regex injected via gatsby-node.js to get the images in the folder for that page.

Quite impressively it succeeds with 8,503 files but starts failing if I add in more.

All the code is here: https://github.com/FraserThompson/dunedinsound.com/tree/gatsby

Expected result

I expected it to be able to build my site because the documentation doesn't mention file limits.

Actual result

The build doesn't succeed.

Environment

gatsby info --clipboard fails for me, probably because of https://github.com/gatsbyjs/gatsby/issues/11496, but this is the output of gatsby info:

System:
OS: Windows 10
CPU: (16) x64 AMD Ryzen 7 2700X Eight-Core Processor
Binaries:
Yarn: 1.13.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
Browsers:
Edge: 44.17763.1.0
npmPackages:
gatsby: ^2.1.4 => 2.1.4
gatsby-image: ^2.0.22 => 2.0.29
gatsby-plugin-feed: ^2.0.8 => 2.0.13
gatsby-plugin-google-analytics: ^2.0.5 => 2.0.14
gatsby-plugin-manifest: ^2.0.5 => 2.0.17
gatsby-plugin-offline: ^2.0.5 => 2.0.23
gatsby-plugin-react-helmet: ^3.0.0 => 3.0.6
gatsby-plugin-sharp: ^2.0.20 => 2.0.20
gatsby-plugin-styled-components: ^3.0.4 => 3.0.5
gatsby-plugin-typography: ^2.2.0 => 2.2.7
gatsby-remark-copy-linked-files: ^2.0.5 => 2.0.9
gatsby-remark-images: ^2.0.4 => 2.0.6
gatsby-remark-prismjs: ^3.0.0 => 3.2.4
gatsby-remark-smartypants: ^2.0.5 => 2.0.8
gatsby-source-filesystem: ^2.0.2 => 2.0.20
gatsby-transformer-remark: ^2.1.15 => 2.2.5
gatsby-transformer-sharp: ^2.1.13 => 2.1.13

[yogeshkotadiya]

I'll try to reproduce and check, this might be caused by the original node fs module.
__EDIT:__ I can't reproduce this because provided repo has a lot of incorrect GraphQL queries. Please fix your graphql queries first.

[FraserThompson]

Sorry, I think the GraphQL queries were failing because I excluded all the images from git since there's gigabytes of them.

I've made a new branch which contains all my original images resized so that they're small enough for source control. It should make it easier to reproduce the issue hopefully.

https://github.com/FraserThompson/dunedinsound.com/tree/gatsby-file-test

[Arystosedes]

I have the same problem, I mean too many open files:

success open and validate gatsby-configs — 0.019 s
success load plugins — 0.710 s
success onPreInit — 1.689 s
success delete html and css files from previous builds — 0.598 s
success initialize cache — 0.019 s
success copy gatsby files — 0.102 s
success onPreBootstrap — 0.020 s
error Plugin gatsby-source-filesystem returned an error

Error: EMFILE: too many open files, open (...)

I have about 8k markdown files to open, and now I'm stuck.. :(

[freiksenet]

Currently we don't have any solutions to this except increasing open file limit (not sure if it's possible on Windows). One workaround could be using CHOKIDAR_USEPOLLING=1 env var to tell chokidar (our fs watcher package) to poll instead of using file watchers. This has performance implications, but could help with this problem.

[FraserThompson]

Thanks for the response. For me CHOKIDAR_USEPOLLING=1 doesn't seem to help, and as far as I know there's no soft file limit on Windows. Maybe I'll look into building it inside a Linux Docker container or something if this is a Windows problem.

Google seems to suggest https://github.com/isaacs/node-graceful-fs as a drop in replacement for fs, I might also experiment with that to see if it makes a difference.

EDIT: I can confirm that monkeypatching fs with graceful-fs at the top of gatsby-node.js as in the snippet below fixes the issue for me.

```javascript
const realFs = require('fs')
const gracefulFs = require('graceful-fs')
gracefulFs.gracefulify(realFs)
````

EDIT2: Actually after upgrading from Node 10 to Node 11 everything seems to be fine without having to patch fs... So all is well!

[yogeshkotadiya]

@FraserThompson Use fs-extra module since it's already added in package.json and also fs-extra is built on top of graceful-fs and it's a drop-in replacement for fs module.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[volkandkaya]

I'm getting this issue with https://github.com/volkandkaya/saaspages

Not a clue what's going on. Changed ulimit and the other things google has recommend me but can't fix it.

I'm using ubuntu.

[dr0i]

@volkandkaya you may have already somehow solved your issue. For me, using the library fs-extra in version 9.0.0 fixed the issue on ubuntu 18.

[adamthrottleup]

I am getting this issue with my gatsby-trasformer-asciidoctor project (over 10k asciidoc files, source is private) running on WSL2 ubuntu 20. @volkandkaya for me the problem is happening in the gatsby-source-filesystem plugin so how do I make it use fs-extra?
Interestingly this only happened when I upgraded my environment from ubuntu 18 on WSL v1 to ubuntu 20 on WSL 2

[FraserThompson]

I should also add that this issue never went away for me. Every time I change Gatsby versions it seems to come back again.

This should be re-opened imo since it seems a bunch of people with large sites are experiencing this issue.

[wamir79]

Hi all,

the best way i found to solve this issue is using graceful-fs.
add in the beginning of gatsby-config.js:

const fs = require('fs');
const gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);

[epbarger]

Also experiencing this issue with gatsby-source-filesystem, only I'm experiencing it on Mac. Agreed that this should be reopened!

The fix wamir79 and others have posted appears to work for me :)