Cli: "EMFILE: too many open files" error with 7.65.3 and 7.65.4

[kidjansen]

Summary

sfdx force:source:retrieve crashes with "EMFILE: too many open files" error

Steps To Reproduce:

Can't share repo (customer IP), but you can contact me if needed for screen sharing session (CET/CEST time zone).

1. sfdx force:source:retrieve on a sandbox or production org for my customer with a large retrieval scope in package.xml

Expected result

retrieval of the audiences in the retrieval scope (package.xml)

Actual result

2020-07-15 12:44:39,607 [2481658]   INFO - .ForceComSfdxMetadataRetriever - Retrieval completed in 18 m 16 s 443 ms. 
2020-07-15 12:44:40,044 [2482095]   WARN - .ForceComSfdxMetadataRetriever - Failed to retrieve metadata.

Status: FAILED
Message: Error
EMFILE: too many open files, open 'C:\Users\kijansen\AppData\Local\Temp\ofc_dev-ofc_dev-retrieve\tmp\force-app\main\default\flowDefinitions\Work_Order_Wait_Email_Handler.flowDefinition-meta.xml'

see idealog.zip for full log generated by IntelliJ + Illuminated Cloud 2.

Additional information

_Feel free to attach a screenshot_.

SFDX CLI Version(to find the version of the CLI engine run sfdx --version):
applicable to both 7.65.3 and 7.65.4, did work in 7.64.x and earlier.

SFDX plugin Version(to find the version of the CLI plugin run sfdx plugins --core)
@oclif/plugin-autocomplete 0.1.5 (core)
@oclif/plugin-commands 1.3.0 (core)
@oclif/plugin-help 3.0.1 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-plugins 1.7.10 (core)
@oclif/plugin-update 1.3.10 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/sfdx-trust 3.0.7 (core)
analytics 1.12.0 (core)
generator 1.1.3 (core)
salesforcedx 48.22.8 (core)
├─ @salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
├─ salesforcedx-templates 48.18.0 (core)
├─ custom-metadata 1.0.4 (core)
└─ salesforce-alm 48.24.1 (core)
sfdx-cli 7.65.4 (core)

OS and version:
Windows 10 Pro with all updates

[kidjansen]

Discussed this with Scott Wells (can't seem to @mention him here), because I thought this was an error in either IntelliJ or Illuminated Cloud 2, but he said the problem is actually in the CLI and asked me to log it here.

[kidjansen]

All issues I've logged regarding 7.65.x are regression by the way, because none of them are applicable anymore after uninstalling the CLI (previously installed with the .msi installer) and reinstall 7.63.0 using the Node.js method of installing.

Other than this issue these are:
https://github.com/forcedotcom/cli/issues/483 (marked as duplicate of https://github.com/forcedotcom/cli/issues/470)
https://github.com/forcedotcom/cli/issues/484 (marked as duplicate of https://github.com/forcedotcom/cli/issues/470)
https://github.com/forcedotcom/cli/issues/485
https://github.com/forcedotcom/cli/issues/486

[DevonBritton]

I'm getting a similar error in Azure DevOps during our build. It's using a Windows hosted agent...

2020-07-28T09:20:11.948Z sfdx:SourceDeployCommand ERROR [ 'ERROR running force:source:deploy: ',
'EMFILE: too many open files, open \'E:\Agents...

It's happening when running a "Command Line" build step with the following command - sfdx force:source:deploy -u $(sf.orgAlias) -p force-app -c

But it's erratic. It's not happening with every build.

Is there any way to mitigate this issue in Azure DevOps or on the Build server itself?

I can provide build log files if necessary.

Kind Regards,
Devon

[kidjansen]

@DevonBritton for us it seemed to be a volume issue. It didn't occur if we significantly limited the scope of deployments or retrievals. Maybe you can deploy in stages, but that might be a challenge.

[DevonBritton]

@DevonBritton for us it seemed to be a volume issue. It didn't occur if we significantly limited the scope of deployments or retrievals. Maybe you can deploy in stages, but that might be a challenge.

Thanks @kidjansen . I thought that was going to be the solution, but it would have been nice if there was something else to be done about it.

[kidjansen]

@DevonBritton if you install the Salesforce CLI with Node.js you can specify which version to install. That way I downgraded to 7.63.0 which works perfectly fine and doesn't have any of the 5 bugs I recently logged.

npm install [email protected] --global

You do get a different path to the CLI in this case than when you install it using the installer (at least on Windows). Normally it's in Program Files, but this way it's in C:\Users\<username>\AppData\Roaming\npm

[aheber]

I have new Windows users experiencing this problem now.

ERROR running force:source:push:  EMFILE: too many open files, open 'C:\Users\user\AppData\Local\Temp\sdx_decomposition_1599849648047\objects\Custom_Object__c.object-meta.xml'

This is while we're pushing our app bundle to a newly created Scratch Org.

@oclif/plugin-autocomplete 0.1.5 (core)
@oclif/plugin-commands 1.3.0 (core)
@oclif/plugin-not-found 1.2.4 (core)
@oclif/plugin-warn-if-update-available 1.7.0 (core)
@oclif/plugin-which 1.0.3 (core)
@salesforce/lwc-dev-server 2.5.1
├─ @oclif/plugin-help 2.2.2
└─ @oclif/plugin-update 1.3.9
@salesforce/sfdx-trust 3.4.3 (core)
alias 1.1.1 (core)
analytics 1.12.1 (core)
config 1.1.8 (core)
generator 1.1.3 (core)
salesforcedx 49.10.0 (core)
├─ templates 49.4.4 (core)
├─ @salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
├─ salesforce-alm 49.11.0 (core)
├─ custom-metadata 1.0.10 (core)
└─ apex 0.0.9 (core)
sfdx-cli 7.74.1 (core)
sfdx-devops 0.3.2
sfdx-heber 0.0.2
sfdx-typegen 0.6.2
shane-sfdx-plugins 4.39.2

[clairebianchi]

We are actively working on a fix for this.

[kidtsunami]

This is an issue with sfdx-cli/7.63.0 win32-x64 node-v12.18.4

[amphro]

I think we missed this one in the release notes and I believe was fixed in 7.76.0 50.1.0. Closing this but please repopen if it isn't fixed in the latest release.

[jeffhube]

@amphro I'm still encountering this error on Windows in the latest release.

Microsoft Windows [Version 10.0.18363.1198]
sfdx-cli/7.82.0-3d0c527ac6 win32-x64 node-v12.18.3

> sfdx force:source:push --json
{
  "status": 1,
  "name": "Error",
  "message": "EMFILE: too many open files, open 'C:\\Users\\****\\AppData\\Local\\Temp\\sdx_decomposition_1607437471364\\objects\\****__c.object-meta.xml'",
  "exitCode": 1,
  "commandName": "SourcePushCommand",
  "stack": "
Error: EMFILE: too many open files, open 'C:\\Users\\****\\AppData\\Local\\Temp\\sdx_decomposition_1607437471364\\objects\\****__c.object-meta.xml'
    at Object.openSync (fs.js:462:3)
    at Proxy.writeFileSync (fs.js:1362:35)
    at AggregateSourceElement.composeMetadata (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\salesforce-alm\\dist\\lib\\source\\aggregateSourceElement.js:461:12)
    at async AggregateSourceElement.getMetadataPathTranslation (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\salesforce-alm\\dist\\lib\\source\\aggregateSourceElement.js:395:67)
    at async AggregateSourceElement.getFilePathTranslations (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\salesforce-alm\\dist\\lib\\source\\aggregateSourceElement.js:375:45)
Outer stack:
    at Function.wrap (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\salesforce-alm\\node_modules\\@salesforce\\core\\lib\\sfdxError.js:171:27)
    at SourcePushCommand.catch (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\salesforce-alm\\dist\\ToolbeltCommand.js:248:46)
    at async SourcePushCommand._run (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\@salesforce\\command\\lib\\sfdxCommand.js:97:13)
    at async Config.runCommand (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\@oclif\\config\\lib\\config.js:173:24)
    at async Main.run (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\@oclif\\command\\lib\\main.js:27:9)
    at async Main._run (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\node_modules\\@oclif\\command\\lib\\command.js:43:20)
    at async Object.run (C:\\Users\\****\\AppData\\Local\\sfdx\\client\\7.82.0-3d0c527ac6\\dist\\cli.js:32:20)",
  "warnings": []
}

(I cleaned up the new lines in the stack trace)

[Praenei]

@amphro Also getting it on Windows 10 (x64) with latest sfdx

sfdx --version
sfdx-cli/7.83.0 win32-x64 node-v8.11.1


sfdx force:source:push -u crm3452 -f --json
(node:17708) Warning: The .forceignore file doesn't adhere to .gitignore format which will be the default behavior starting in Spring '21 release. More information on .gitignore format here: https://git-scm.com/docs/gitignore. Fix the following lines in your .forceignore and add '# .forceignore v2' 
to your .forceignore file to switch to the new behavior.
(node:17708) Warning:   force-app/main/default/applications/Live_Chat
{
  "status": 1,
  "name": "Error",
  "message": "EMFILE: too many open files, open 'C:\\Users\\*****\\AppData\\Local\\Temp\\1\\sdx_decomposition_1607610999129\\objects\\CollaborationGroup.object-meta.xml'",
  "exitCode": 1,
  "commandName": "SourcePushCommand",
  "stack": "Error: EMFILE: too many open files, open 'C:\\Users\\*****\\AppData\\Local\\Temp\\1\\sdx_decomposition_1607610999129\\objects\\CollaborationGroup.object-meta.xml'\n    at Object.fs.openSync (fs.js:646:18)\n    at Proxy.fs.writeFileSync (fs.js:1299:33)\n    at AggregateSourceElement.composeMetadata (C:\\Users\\*****\\AppData\\Roaming\\npm\\node_modules\\sfdx-cli\\node_modules\\salesforce-alm\\dist\\lib\\source\\aggregateSourceElement.js:461:12)\n    at <anonymous>\nOuter stack:\n    at Function.wrap (C:\\Users\\*****\\AppData\\Roaming\\npm\\node_modules\\sfdx-cli\\node_modules\\@salesforce\\core\\lib\\sfdxError.js:171:27)\n    at SourcePushCommand.catch (C:\\Users\\*****\\AppData\\Roaming\\npm\\node_modules\\sfdx-cli\\node_modules\\salesforce-alm\\dist\\ToolbeltCommand.js:248:46)\n    at <anonymous>",
  "warnings": []
}

[AnamikaRathoreJandu]

@Praenei @jeffhube I am getting the same issue on _sfdx push_ for the version sfdx-cli/7.82.0-52018670a1 win32-x64 node-v12.18.3 and unable to downgrade it to the previous version. Did you find a fix for it? If so, could you please tell me the workaround? Thanks :)

[alexwebck]

The same error has been encountered to me for the version _sfdx-cli/7.82.0-3d0c527ac6 win32-x64 node-v12.18.3_

[Praenei]

@AnamikaRathoreJandu - sorry but no workaround. Oddly we have around 12 people with the same set up & seems to be working for around half. Previously I've downgraded to 7.61 (which works) but after Winter 21 that version has other issues for us

[geekishor]

We are actively working on a fix for this.

Any update?

[AnamikaRathoreJandu]

@AnamikaRathoreJandu - sorry but no workaround. Oddly we have around 12 people with the same set up & seems to be working for around half. Previously I've downgraded to 7.61 (which works) but after Winter 21 that version has other issues for us

We've managed to fix it by adding our test classes folder to forceignore temporarily, downgrading SFDX to 47 (since our metadata is on API version 47) and then attempting to push again. I think there's an issue downgrading to 49 (it doesn't downgrade to 49 at all) but I'm sure up until 48 should work.
Try forceignoring the largest chunk(s) of your source code - for us it was tests.
Hope that helps :)

[shetzel]

The fix will be in the first patch release of 2021 (early January)

[ssoundr]

Is patch for this issue provided?

We are facing the issue in Linux env using the latest sfdx
sfdx --version
sfdx-cli/7.82.0-52018670a1 linux-x64 node-v12.18.3

"stack": "Error: EMFILE: too many open files, open '/tmp/sdx_decomposition_1610418910106/objects/*.object-meta.xml'\n at Object.openSync (fs.js:462:3)\n at Proxy.writeFileSync (fs.js:1362:35)\n at AggregateSourceElement.composeMetadata (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:461:12)\n at async AggregateSourceElement.getMetadataPathTranslation (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:395:67)\n at async AggregateSourceElement.getFilePathTranslations (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:375:45)\nOuter stack:\n at Function.wrap (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/@salesforce/core/lib/sfdxError.js:171:27)\n at SourceDeployCommand.catch (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/ToolbeltCommand.js:248:46)\n at async SourceDeployCommand._run (/usr/local/lib/sfdx/node_modules/@salesforce/command/lib/sfdxCommand.js:97:13)\n at async Config.runCommand (/usr/local/lib/sfdx/node_modules/@oclif/config/lib/config.js:173:24)\n at async Main.run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/main.js:27:9)\n at async Main._run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/command.js:43:20)\n at async Object.run (/usr/local/lib/sfdx/dist/cli.js:32:20)",

[geekishor]

Is patch for this issue provided?

We are facing the issue in Linux env using the latest sfdx
sfdx --version
sfdx-cli/7.82.0-52018670a1 linux-x64 node-v12.18.3

"stack": "Error: EMFILE: too many open files, open '/tmp/sdx_decomposition_1610418910106/objects/*.object-meta.xml'\n at Object.openSync (fs.js:462:3)\n at Proxy.writeFileSync (fs.js:1362:35)\n at AggregateSourceElement.composeMetadata (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:461:12)\n at async AggregateSourceElement.getMetadataPathTranslation (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:395:67)\n at async AggregateSourceElement.getFilePathTranslations (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/lib/source/aggregateSourceElement.js:375:45)\nOuter stack:\n at Function.wrap (/usr/local/lib/sfdx/node_modules/salesforce-alm/node_modules/@salesforce/core/lib/sfdxError.js:171:27)\n at SourceDeployCommand.catch (/usr/local/lib/sfdx/node_modules/salesforce-alm/dist/ToolbeltCommand.js:248:46)\n at async SourceDeployCommand._run (/usr/local/lib/sfdx/node_modules/@salesforce/command/lib/sfdxCommand.js:97:13)\n at async Config.runCommand (/usr/local/lib/sfdx/node_modules/@oclif/config/lib/config.js:173:24)\n at async Main.run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/main.js:27:9)\n at async Main._run (/usr/local/lib/sfdx/node_modules/@oclif/command/lib/command.js:43:20)\n at async Object.run (/usr/local/lib/sfdx/dist/cli.js:32:20)",

Try with 7.70.1 version. I am able to deploy with this version.
sfdx-cli/7.70.1 win32-x64 node-v15.4.0

[jairzh]

We got the same issue when using force:source:push to push source code with Platform Event object.

ERROR running force:source:push: EMFILE: too many open files, open '/tmp/sdx_decomposition_1610596367395/objects/TestEvent__e.object-meta.xml'

Version information:

salesforcedx 50.7.0 (core)
├─ schema 1.0.1 (core)
├─ salesforce-alm 50.7.0 (core)
├─ templates 50.1.0 (core)
├─ @salesforce/sfdx-plugin-lwc-test 0.1.7 (core)
├─ custom-metadata 1.0.10 (core)
└─ apex 0.1.4 (core)
sfdx-cli 7.82.0 (core)

[amphro]

These are incredibly hard to reproduce. We tried to fix all of them by updating the fs library that is supposed to prevent these types of errors. With that said, it looks like there are areas we missed. Please file a new issue with detailed repro steps including a link to a gist with --dev-debug output. If you can share a repository that can reproduce this, it will increase the chances that we are able to fix it.