Copy-webpack-plugin: "Error: EMFILE: too many open files, open.." on Windows

Created on 21 Jun 2016 · 23Comments · Source: webpack-contrib/copy-webpack-plugin

That's pretty much the issue.
Works fine on OS X

Have googled and can't find any way to increase Windows open file limit.
Is now also happening on OS X 'ENFILE: file table overflow'

As far as I can tell this module always works down to an individual file level in async.

I've worked around this by changing the app structure (it will be the ~8000 files in node_modules) and doing a shell cp instead.

Is it something you need to look at handling via something like graceful-fs?

Bug

Source

TimNZ

👍3

Most helpful comment

This is tricky, because node-glob no longer uses graceful-fs and just including it before node-glob won't work because graceful-fs has stopped monkey-patching fs. The graceful-fs docs say not to patch fs in a library (like this one), so it'd have to be added as an option. The problem is, the option would have to be checked before any other module loaded fs. An environment variable is the only reasonable way I can think to do that, which seems sloppy.

If you're running into this issue, you'll need to monkey-patch fs yourself. Here's how:

// At the top of your webpack config, insert this
var fs = require('fs');
var gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);

I'd add this info to the docs.

kevlened on 25 Jun 2016

👍13

All 23 comments

I have a same problem.

qiaolb on 24 Jun 2016

If you're running into this issue, you'll need to monkey-patch fs yourself. Here's how:

// At the top of your webpack config, insert this
var fs = require('fs');
var gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);

I'd add this info to the docs.

kevlened on 25 Jun 2016

👍13

The irony of me suggesting graceful-fs based on comments I real elsewhere for different issues and then never looking at the module myself and seeing that it would help :)

TimNZ on 26 Jun 2016

No worries! Others had the same issue, so I'm glad it's in the docs now.

kevlened on 26 Jun 2016

@kevlened i have same issues,i already monkey-patch fs,below is my code:

var fs = require('fs');
var gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);
const webpackMerge = require('webpack-merge');
const ExtractTextPlugin = require('extract-text-webpack-plugin');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const commonConfig = require('./webpack.common.js');
const helpers = require('./helpers');

module.exports = webpackMerge(commonConfig, {
  ......
plugins: [
    new CopyWebpackPlugin([
      { from: helpers.root('node_modules/ckeditor/ckeditor.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/config.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/styles.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/contents.css'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/skins/moono'), to: 'node_modules/ckeditor/skins/moono' },
      { from: helpers.root('node_modules/ckeditor/lang/zh-cn.js'), to: 'node_modules/ckeditor/lang' },
      { from: helpers.root('node_modules/ckeditor/plugins'), to: 'node_modules/ckeditor/plugins' },
      { from: helpers.root('node_modules/ckeditor/adapters'), to: 'node_modules/ckeditor/adapters' },
    ])
  ]
}

error msg:

95% emittingfs.js:634
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: EMFILE: too many open files, open 'D:\work\nodejs\angular\angular2-electron\node_modules\ckeditor\plugins\sourcedialog\lang\ar.js'
    at Error (native)
    at Object.fs.openSync (fs.js:634:18)
    at Object.fs.readFileSync (fs.js:502:33)
    at Object.source (D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\copy-webpack-plugin\3.0.1\copy-webpack-plugin\dist\writeFileToAssets.js:45:31)
    at Compiler.writeOut (D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\webpack\2.1.0-beta.15\webpack\lib\Compiler.js:290:26)
    at D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\mkdirp\0.5.1\mkdirp\index.js:48:26
    at FSReqWrap.oncomplete (fs.js:117:15)

keyiis on 5 Jul 2016

@kevlened I'm seeing the same issue that @keyiis is reporting. Even after applying this patch, the error still occurs when trying to copy the "ckeditor" folder.

kb3eua on 27 Jul 2016

jgrosso on 29 Jul 2016

@kb3eua I use gulp to copy file.

keyiis on 30 Jul 2016

@keyiis I didn't want to introduce another task runner. I was able to get it working by upgrading to the latest version of this plugin and using the following configuration to ignore certain files:

 new CopyWebpackPlugin([{
    from: './node_modules/ckeditor',
    to: './dist/scripts/ckeditor',
    ignore: [
        '**/samples/**',
        '**/lang/!(en.js)'
    ]
}]);

kb3eua on 1 Aug 2016

😄2 👍2 🎉1

Having the same error on Windows with many files (Angular 2 CLI Webpack ng build, bower_components in assets; yeah, I know). graceful-fs does not help, most likely because it does not patch readFileSync/openSync codepath. Webpack's Compiler.js emitFiles uses unconstrained async.ForEach over assets, hitting the Node limit on open files (2048 on Windows, due to use of CRT).

require("async").forEach(Object.keys(compilation.assets), function(file, callback) {

Workaround: I don't understand webpack/copy-webpack-plugin/graceful-fs well enough to suggest the right way of fixing this, but constraining webpack to a reasonable number of async operations (using forEachLimit) works fine:

require("async").forEachLimit(Object.keys(compilation.assets), 2000,
  require("async").ensureAsync(function(file, callback) {

EDIT: Wrapped callback with async.ensureAsync - fixes stack overflow for when assets are served from memory

tallaxes on 20 Sep 2016

having this issue on OSX (patch applied and all)

id0Sch on 21 Sep 2016

This is still very much an issue, the monkey patch does not work.
Can we get this issue reopened for visibility @kevlened.

Ive looked into it, but am still clueless as to a fix.

To add a bit more detail, this is the exact error Im getting. I have add issues across Windows, OSX and Linux.

5752ms optimize chunk assets
1773ms optimize assets
95% emitfs.js:634
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^

Error: EMFILE: too many open files, open '/home/mcp/Sources/saltim-desktop/src/node_modules/lodash/_baseIsEqual.js'
at Error (native)
at Object.fs.openSync (fs.js:634:18)
at Object.fs.readFileSync (fs.js:502:33)
at Object.source (/home/mcp/Sources/saltim-desktop/node_modules/copy-webpack-plugin/dist/writeFileToAssets.js:45:31)
at Compiler.writeOut (/home/mcp/Sources/saltim-desktop/node_modules/webpack/lib/Compiler.js:256:26)
at /home/mcp/Sources/saltim-desktop/node_modules/mkdirp/index.js:48:26
at FSReqWrap.oncomplete (fs.js:117:15)

npm ERR! Linux 4.4.0-42-generic

Worst of all, I simply doe a cp before or after webpack runs, it appears my build doesn't pick up the folder, so thats not a solution for me unfortunately.

McPo on 19 Oct 2016

@tallaxes path does work, although I needed a much lower limit (100). Patching WebPack itself also isn't ideal.

The file that needs to be patched is ./node_modules/webpack/lib/Compiler.js.
Make sure not to forget the closing bracket.

Thanks @tallaxes, Im still looking for a more elegant solution though. I doubt Webpack themselves will accept a PR when the number is variable.

McPo on 19 Oct 2016

@McPo Webpack: It is still worth bringing it to their attention, I think. And there could be ways of coming up with a reasonable number, or just making it configurable.

100 seems unreasonably low; wonder what would limit it like that... For OSX/Linux: did you check OS-level file descriptor limits (i.e. ulimit -n)?

tallaxes on 19 Oct 2016

@tallaxes done.
https://github.com/webpack/webpack/issues/3164

Ive bumped it back up to 1000 it works fine. I suppose my machine is under heavy load, having said that, lowering it to 100 had no adverse effect on the build time.

McPo on 19 Oct 2016

My issues looks like the floder has symlink,resulting in a circular dependency,but I'm not sure.

keyiis on 20 Oct 2016

Ive created a PR. for those suffering from this issue you can use it right away by setting your package.json to

"webpack": "McPo/webpack#1.13.2-file-limit"

Ive limited to 1000, but perhaps 500 is more reasonable.
You can to set it to a user defined value using the environment variable WEBPACK_FILE_LIMIT

https://github.com/webpack/webpack/pull/3166

McPo on 20 Oct 2016

The upcoming version of copy-webpack-plugin (currently in master) avoids fs.readFileSync, which is the main source of the issue, because it isn't patched by graceful-fs. That should allow graceful-fs to be a proper solution.

For additional control, I've added a concurrency option that will allow changing how many files the plugin attempts to read at once (currently set to a default of 1000, but it sounds like that should be reduced). Does 100 sound like a reasonable default?

kevlened on 20 Oct 2016

This should be fixed in the latest version (v4.0.0). We avoid fs.readFileSync, so graceful-fs can now patch the issue, but there are also extra measures to ensure we don't read too many files at once (a concurrency option, which defaults to 100). I'm closing this for now, but if you encounter the error after upgrading to v4.0.0, I'll reopen.

kevlened on 23 Oct 2016

still get this error on 4.0.1