Yarn: mkdirp causes an out of memory error when making a directory to a Windows drive that doesn't exist
Reporting a bug
What is the current behavior?
yarn when issued any command stalls forever until it reports
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Even by setting $ENV:NODE_COMMANDS="--max-old-space-size=4096", yarn will just consume it all and then fail.
If the current behavior is a bug, please provide the steps to reproduce.
This was a fresh install of windows 10, I had recently just completed the build tool installation for NodeJS, and then installed yarn 1.12. I opened up a powershell window, then typed in yarn config list.
What is the expected behavior?
Anything to work
Please mention your node.js, yarn and operating system version.
Windows 10 17134.345node v10.12.0with build toolsyarn 1.12AND1.10.1
Everspace
All 17 comments
This is now occurring on my v1.10.1 install as well.
The logs look like this
PS S:\git\Excellent-Homebrew-Approach> yarn
yarn install v1.10.1
<--- Last few GCs --->
[9800:000001D3A2A792E0] 84590 ms: Scavenge 1393.2 (1423.2) -> 1392.4 (1423.7) MB, 5.3 / 0.0 ms (average mu = 0.117, current mu = 0.040) allocation failure
[9800:000001D3A2A792E0] 84626 ms: Scavenge 1393.6 (1423.7) -> 1393.0 (1424.7) MB, 6.3 / 0.0 ms (average mu = 0.117, current mu = 0.040) allocation failure
[9800:000001D3A2A792E0] 84664 ms: Scavenge 1394.3 (1424.7) -> 1393.6 (1429.7) MB, 6.4 / 0.0 ms (average mu = 0.117, current mu = 0.040) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF610A0ACB5
2: 00007FF6109E44E6
3: 00007FF6109E4EF0
4: 00007FF610C747FE
5: 00007FF610C7472F
6: 00007FF6111A8964
7: 00007FF61119F0F2
8: 00007FF61119D68C
9: 00007FF6111A37E7
10: 00007FF61102AEBC
11: 00007FF610967A5E
12: 00007FF610968053
13: 00007FF610A4EBE4
14: 00007FF610A4F7F3
15: 00007FF6109EB85E
16: 00007FF6109EC135
17: 00007FF6109ECA11
18: 00007FF6108996C9
19: 00007FF611476248
20: 00007FFC5F4D3034
21: 00007FFC5FF71461
npm install works fine, so I'm not suspecting my current setup of nodejs.
Everspace
on 13 Oct 2018
I'm afraid that's the kind of issue that we'll have to ask you to investigate by yourself. It doesn't repro anywhere, so there has to be an environmental factor somewhere.
That doesn't mean Yarn isn't potentially doing something wrong, just that we can't find it without your help
arcanis
on 13 Oct 2018
Very reasonable, although personally I wouldn't even know where to start.
Everspace
on 13 Oct 2018
Usually when I have issues like this I try to add console.log and process.exit statements a bit everywhere to figure out in which part of the program are things crashing. I think a good start would be to add some in the class Install definitions, in particular within the init function, and even more in particular the various callbacks associated with the steps.push calls.
arcanis
on 13 Oct 2018
It fails before the init.
Placing something like console.log("initted") at this point shows nothing happening.
https://github.com/yarnpkg/yarn/blob/aa4e713c30c40ac35934e45aca5b4683d61e1829/src/cli/index.js#L532
It seems like await config.init() is the problem area.
Everspace
on 13 Oct 2018
Strange - then the problem has to come from somewhere around here, but afaik this part of the code is no reentrant, so I don't see anything obvious that could cause a stack overflow 🤔
https://github.com/yarnpkg/yarn/blob/aa4e713c30c40ac35934e45aca5b4683d61e1829/src/config.js#L253-L447
arcanis
on 13 Oct 2018
I think the issue is the following:
.yarnrc
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
cache-folder "S:\\yarn-cache"
lastUpdateCheck 1539138928914
The S: drive on my machine doesn't exist. This happened when I reinstalled and my letters got mangled, but I ported over all my old dotfiles.
I think what is happening is that it is trying to do something with the yarn-cache, and doesn't handle the rejection of a drive not existing, spinning out of control until it explodes.
Changing the cache-folder to an existing drive resolved the issue.
Everspace
on 13 Oct 2018
I found this by turning on verbose, and going through the configuration file steps since that was the last area that was displayed.
Everspace
on 13 Oct 2018
It is this block that attempts to make a directory to yarn-cache
This seems to be 4-year-old problem with mkdirp and non-existent drives https://github.com/substack/node-mkdirp/issues/66
Everspace
on 14 Oct 2018
@arcanis I'm not sure how yarn wants to handle this problem with the dependency. There was a pull request, but it seems like mkdirp is not maintained anymore so that fix is not happening.
Everspace
on 16 Oct 2018
Maybe it would make sense to switch to fs-extra instead of mkdirp and rimraf. It would probably have a bad impact on the size, though :/
arcanis
on 16 Oct 2018
make-dir is an option. It looks actively maintained, and doesn't have this problem.
Gudahtt
on 16 Oct 2018
If nothing else, there could be something like a check around fs.mkdirp that abandons ship before this case is hit.
Everspace
on 17 Oct 2018
I'm not sure if this is the same issue, but I'm currently running into this issue with two different projects. In the "Linking dependencies" step it runs up to almost 3GB before running out of memory:
$ yarn add typescript @types/node @types/react @types/react-dom @types/jest
yarn add v1.10.1
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has incorrect peer dependency "react@^15.0.0".
warning " > [email protected]" has incorrect peer dependency "react-dom@^15.0.0".
warning " > @storybook/[email protected]" has unmet peer dependency "@storybook/addons@^3.3.0".
warning " > @storybook/[email protected]" has unmet peer dependency "babel-core@^6.26.0 || ^7.0.0-0".
warning " > @storybook/[email protected]" has unmet peer dependency "babel-runtime@>=6.0.0".
warning "@storybook/react > [email protected]" has unmet peer dependency "babel-core@6".
<--- Last few GCs --->
[31612:0x2756e50] 277231 ms: Mark-sweep 2875.2 (2976.1) -> 2874.7 (2977.6) MB, 369.1 / 0.0 ms allocation failure GC in old space requested
[31612:0x2756e50] 277609 ms: Mark-sweep 2874.7 (2977.6) -> 2874.6 (2944.1) MB, 376.6 / 0.0 ms last resort GC in old space requested
[31612:0x2756e50] 277959 ms: Mark-sweep 2874.6 (2944.1) -> 2874.6 (2944.1) MB, 348.3 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x35d531fa5879 <JSObject>
1: set(this=0x1cec460dd5c9 <Map map = 0xc712e1848d9>,0xc6d2aee4011 <Very long string[3061]>,0x37d3b4d731f9 <HoistManifest map = 0x31e37cf8abd9>)
2: declareRename [/usr/share/yarn/lib/cli.js:~84700] [pc=0x18f9f5e95bef](this=0x2d47fad022b9 <PackageHoister map = 0x31e37cfb85a1>,info=0x37d3b4d731f9 <HoistManifest map = 0x31e37cf8abd9>,oldParts=0x37d3b4d719b9 <JSArray[167]>,newParts=0x1df0829...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x8c20ec [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [node]
8: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashMap>) [node]
9: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x18f9f5c042fd
[1] 31612 abort (core dumped) yarn add typescript @types/node @types/react @types/react-dom @types/jest
I don't have a custom cache folder in my .yarnrc though. I do use Create React App, with TypeScript support, which I can imagine includes quite a few dependencies - not sure if this should then be expected though?
$ yarn --version
1.10.1
$ node --version
v8.11.2
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
Vinnl
on 30 Oct 2018
The issue reported here is about Windows drive, so yours sounds unrelated.
Can you open a new thread? 🙂
Since we likely won't be able to reproduce it, one of the thing that would
be helpful would be for you to tell us in which part of the code does the
memory exhaustion happen (cf previous posts for some advices as to how to
do so).
On Tue, Oct 30, 2018, 4:34 AM Vincent notifications@github.com wrote:
I'm not sure if this is the same issue, but I'm currently running into
this issue with two different projects. In the "Linking dependencies" step
it runs up to almost 3GB before running out of memory:$ yarn add typescript @types/node @types/react @types/react-dom @types/jest
yarn add v1.10.1
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > [email protected]" has incorrect peer dependency "react@^15.0.0".
warning " > [email protected]" has incorrect peer dependency "react-dom@^15.0.0".
warning " > @storybook/[email protected]" has unmet peer dependency "@storybook/addons@^3.3.0".
warning " > @storybook/[email protected]" has unmet peer dependency "babel-core@^6.26.0 || ^7.0.0-0".
warning " > @storybook/[email protected]" has unmet peer dependency "babel-runtime@>=6.0.0".
warning "@storybook/react > [email protected]" has unmet peer dependency "babel-core@6".<--- Last few GCs --->
[31612:0x2756e50] 277231 ms: Mark-sweep 2875.2 (2976.1) -> 2874.7 (2977.6) MB, 369.1 / 0.0 ms allocation failure GC in old space requested
[31612:0x2756e50] 277609 ms: Mark-sweep 2874.7 (2977.6) -> 2874.6 (2944.1) MB, 376.6 / 0.0 ms last resort GC in old space requested
[31612:0x2756e50] 277959 ms: Mark-sweep 2874.6 (2944.1) -> 2874.6 (2944.1) MB, 348.3 / 0.0 ms last resort GC in old space requested<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x35d531fa5879
1: set(this=0x1cec460dd5c9
arcanis
on 30 Oct 2018
@arcanis Right, do you know where the Yarn code is installed when I used the Debian repository?
That said, I just thought of a dependency with a lot of subdependencies that could be removed, and now it works again. Is it still interesting to report a new issue?
Vinnl
on 30 Oct 2018
Related issues
NonPolynomial
·
3Comments
mnpenner
·
3Comments
chiedo
·
3Comments
danez
·
3Comments
baptistelebail
·
3Comments
Most helpful comment
I think the issue is the following:
.yarnrcThe S: drive on my machine doesn't exist. This happened when I reinstalled and my letters got mangled, but I ported over all my old dotfiles.
I think what is happening is that it is trying to do something with the
yarn-cache, and doesn't handle the rejection of a drive not existing, spinning out of control until it explodes.Changing the
cache-folderto an existing drive resolved the issue.