Yarn: Memory optimization

Created on 22 Aug 2016  ·  20Comments  ·  Source: yarnpkg/yarn

Trying to use [email protected] in install https://github.com/yunxing/jengaboot (with about 200 dependencies) and it OOMed.

I've tried to increase the mem limit to 8GB, but still no luck.

#!/usr/bin/env node --max-old-space-size=8192

~/o/jengaboot (master|✔) $ /usr/local/bin/kpm install
kpm install v0.10.0
info No lockfile found.
[1/3] 🚚  Resolving and fetching packages...
[2/3] 🔗  Linking dependencies...
<--- Last few GCs --->

  520381 ms: Scavenge 8186.6 (8343.1) -> 8186.8 (8343.1) MB, 3.3 / 0 ms (+ 7.6 ms in 2 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  525296 ms: Mark-sweep 8186.8 (8343.1) -> 8182.9 (8343.1) MB, 4909.0 / 0 ms (+ 251.5 ms in 2173 steps since start of marking, biggest step 10.0 ms) [last resort gc].
  529578 ms: Mark-sweep 8182.9 (8343.1) -> 8177.2 (8343.1) MB, 4282.6 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2b007e0e3ac1 <JS Object>
    1: add [native collection.js:~113] [pc=0x394f0e327998] (this=0xaf3ab1f20c1 <a Set with map 0x28d770308bf1>,m=0xc5ee9706909 <JS Array[2]>)
    2: addTransitive [/Users/yunxing/kpm/dist/lib/PackageHoister.js:~48] [pc=0x394f0e328b94] (this=0xaf3ab1f2051 <a HoistManifest with map 0x2a99f52a3539>,pairs=0x2a2986466451 <JS Array[52527]>)
    3: _seed [/Users/yunxing/kpm/dist/lib/PackageHoister.js:...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
fish: Job 1, '/usr/local/bin/kpm install' terminated by signal SIGABRT (Abort)
picture yunxing

All 20 comments

Fixed this, I'll put up a PR in a bit. Looks like there was some pathological case with the dependency combo where we were recursively creating entries in our package hoister.

sebmck picture sebmck on 22 Aug 2016
👍1

I'm getting:

kpm install v0.9.0
info No lockfile found.
[1/3] 🚚  Resolving and fetching packages...
[2/3] 🔗  Linking dependencies...
[3/3] 📃  Building fresh packages...
error "sh -c eval $(dependencyEnv) && nopam && ocaml setup.ml -configure --prefix $opam_prefix && ocaml setup.ml -build && ocaml setup.ml -install && (opam-installer --prefix=$opam_prefix || true) && injectOpamEnvs"@/Users/sebmck/Scratch/jengaboot/node_modules/lambda-term: ocamlfind: Package `lwt.unix' not found
W: Field 'pkg_lwt_unix' is not set: Command ''/Users/sebmck/Scratch/jengaboot/node_modules/ocamlfind/_build/ocamlfind/bin/ocamlfind' query -format %d lwt.unix > '/var/folders/cf/4qcm5sp53nvb7x6cpngtf18jjrkxkh/T/oasis-89f5a4.txt'' terminated with error code 2
ocamlfind: Package `lwt.react' not found
W: Field 'pkg_lwt_react' is not set: Command ''/Users/sebmck/Scratch/jengaboot/node_modules/ocamlfind/_build/ocamlfind/bin/ocamlfind' query -format %d lwt.react > '/var/folders/cf/4qcm5sp53nvb7x6cpngtf18jjrkxkh/T/oasis-ba27d2.txt'' terminated with error code 2
E: Cannot find findlib package lwt.react
E: Cannot find findlib package lwt.unix
E: Failure("2 configuration errors")
    at ChildProcess.<anonymous> (/Users/sebmck/Projects/fbkpm/lib/util/child.js:104:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:821:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)

Not sure if this is a bug in kpm or my local environment? Any ideas?

sebmck picture sebmck on 23 Aug 2016

Haven't got to step 3 before due to the memory issue.

My theory about this issue:

  • We use a npm package called dependencyEnv to expose whether a package is installed or not via environment variable.

    • DependencyEnv checks "dependencies" and "peerDependencies" (note, not "optionaldependencies") to find dependencies of a package, and read the environment variables exposed by the those dependencies.

    • Unix and React are optional dependencies of package lwt.

    • since the field "optionalDepenencies" are not considered as the dependencies of a package, lwt is build without lwt.React and lwt.unix.

    • lambda-term failed to be built because it cannot find lwt.react and lwt.unix.

The above method somehow works with npm because npm copies "optionalDependencies" to "dependencies" during the package resolution step. Therefore "react" and "unix" are considered dependencies of lwt in npm, but not in kpm.

yunxing picture yunxing on 23 Aug 2016

Overall I think the model of "optionalDependencies" is broken as it makes a package build non-deterministic. I'm going to remove "optionalDependencies" and list them as "dependencies" instead for our packages. Hopefully it can resolve the issue here.

yunxing picture yunxing on 23 Aug 2016

@kittens Try it again with a clean cache? I've just updated the packages.

yunxing picture yunxing on 23 Aug 2016

Got much further this time!

kpm install v0.9.0
info No lockfile found.
[1/3] 🚚  Resolving and fetching packages...
[2/3] 🔗  Linking dependencies...
[3/3] 📃  Building fresh packages...
error "sh -c eval $(dependencyEnv) && nopam && substs pkg/META.in && make compile_error && ocaml pkg/build.ml native=true native-dynlink=true utop=${utop_installed:-false} && (opam-installer --prefix=$opam_prefix || true)"@/Users/sebmck/Scratch/jengaboot/node_modules/reason: Read 831 sample input sentences and 831 error messages.
Read 831 sample input sentences and 831 error messages.
    at ChildProcess.<anonymous> (/Users/sebmck/Projects/fbkpm/lib/util/child.js:104:15)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:821:16)
    at Socket.<anonymous> (internal/child_process.js:319:11)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at Pipe._onclose (net.js:469:12)
sebmck picture sebmck on 23 Aug 2016

hmm the message "Read 831 sample input sentences and 831 error messages." is actually expected (it generates error messages based on compiler states). Don't know why the command failed though.

yunxing picture yunxing on 23 Aug 2016

I'll include stdout in the error message, should help hopefully.

sebmck picture sebmck on 23 Aug 2016
👍1

Not getting much info unfortunately 😞 

Exit code: 10
Command: sh -c eval $(dependencyEnv) && nopam && substs pkg/META.in && make compile_error && ocaml pkg/build.ml native=true native-dynlink=true utop=${utop_installed:-false} && (opam-installer --prefix=$opam_prefix || true)
Directory: /Users/sebmck/Scratch/jengaboot/node_modules/reason
Output: Read 831 sample input sentences and 831 error messages.

Exit code 10 is what's causing it to be registered as a failure, is that right?

sebmck picture sebmck on 23 Aug 2016

@kittens right.

Hard to debug without stdout. Don't want to drag you too deep into this. Do you mind to publish what you have so far and I can take over the debugging?

yunxing picture yunxing on 23 Aug 2016

No problem, sorry for the back and forth. The output in the last comment should include stdout too. I've pushed what I have to the branch publish which is PR #235.

sebmck picture sebmck on 23 Aug 2016
👍1

@kittens Thanks. Let me tweak a bit to see what I can find.

yunxing picture yunxing on 23 Aug 2016

Oops, this is the output that includes stdout:

error Command failed.
Exit code: 10
Command: sh
Arguments: -c eval $(dependencyEnv) && nopam && substs pkg/META.in && make compile_error && ocaml pkg/build.ml native=true native-dynlink=true utop=${utop_installed:-false} && (opam-installer --prefix=$opam_prefix || true)
Directory: /Users/sebmck/Scratch/jengaboot/node_modules/reason
Output:
Read 831 sample input sentences and 831 error messages.
The auto-generated comments in src/reason_parser.messages have been re-generated. The old messages file has been backed up at src/reason_parser.messages.bak
echo "" > src/reason_parser.messages.simplified
echo "## DO NOT MODIFY -- auto generated error message from reason_parser.messages" >> src/reason_parser.messages.simplified
echo "## See https://github.com/facebook/reason/issues/611" >> src/reason_parser.messages.simplified
echo "" >> src/reason_parser.messages.simplified
cat src/reason_parser.messages | grep -v "##" >> src/reason_parser.messages.simplified
menhir --explain --strict --unused-tokens src/reason_parser.mly --compile-errors src/reason_parser.messages > src/reason_parser_message.ml
Read 831 sample input sentences and 831 error messages.
ocamlfind ocamlopt unix.cmxa -I /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa -linkpkg myocamlbuild.ml /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
ocamlfind ocamldep -package 'menhirLib unix compiler-libs.common ocamlbuild findlib easy-format BetterErrors merlin_extend re.str' -modules src/reasonbuild.ml > src/reasonbuild.ml.depends
ocamlfind ocamlc -c -I +ocamldoc -g -bin-annot -safe-string -package 'menhirLib unix compiler-libs.common ocamlbuild findlib easy-format BetterErrors merlin_extend re.str' -w @5@8@10@11@12@14@23-24@26@29@40 -I src -o src/reasonbuild.cmo src/reasonbuild.ml
+ ocamlfind ocamlc -c -I +ocamldoc -g -bin-annot -safe-string -package 'menhirLib unix compiler-libs.common ocamlbuild findlib easy-format BetterErrors merlin_extend re.str' -w @5@8@10@11@12@14@23-24@26@29@40 -I src -o src/reasonbuild.cmo src/reasonbuild.ml
findlib: [WARNING] Package ocamlbuild has multiple definitions in /Users/sebmck/Scratch/jengaboot/node_modules/ocamlbuild/_build/ocamlfind/lib/ocamlbuild/META, /Users/sebmck/Scratch/jengaboot/node_modules/ocamlfind/_build/ocamlfind/lib/ocamlbuild/META
findlib: [WARNING] Interface topdirs.cmi occurs in several directories: /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/compiler-libs, /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml
ocamlfind ocamlopt -c -I +ocamldoc -g -bin-annot -safe-string -package 'menhirLib unix compiler-libs.common ocamlbuild findlib easy-format BetterErrors merlin_extend re.str' -w @5@8@10@11@12@14@23-24@26@29@40 -I src -o src/reasonbuild.cmx src/reasonbuild.ml
+ ocamlfind ocamlopt -c -I +ocamldoc -g -bin-annot -safe-string -package 'menhirLib unix compiler-libs.common ocamlbuild findlib easy-format BetterErrors merlin_extend re.str' -w @5@8@10@11@12@14@23-24@26@29@40 -I src -o src/reasonbuild.cmx src/reasonbuild.ml
findlib: [WARNING] Package ocamlbuild has multiple definitions in /Users/sebmck/Scratch/jengaboot/node_modules/ocamlbuild/_build/ocamlfind/lib/ocamlbuild/META, /Users/sebmck/Scratch/jengaboot/node_modules/ocamlfind/_build/ocamlfind/lib/ocamlbuild/META
findlib: [WARNING] Interface topdirs.cmi occurs in several directories: /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/compiler-libs, /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml
mkdir -p _reasonbuild; cd _reasonbuild; pwd; touch myocamlbuild.ml; chmod +x ../src/reopt.sh; ocamlbuild -just-plugin -ocamlopt 'env REASON_BUILD_DIR=../../src ../../src/reopt.sh'
+ mkdir -p _reasonbuild; cd _reasonbuild; pwd; touch myocamlbuild.ml; chmod +x ../src/reopt.sh; ocamlbuild -just-plugin -ocamlopt 'env REASON_BUILD_DIR=../../src ../../src/reopt.sh'
/Users/sebmck/Scratch/jengaboot/node_modules/reason/_build/_reasonbuild
env REASON_BUILD_DIR=../../src ../../src/reopt.sh unix.cmxa -I /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa myocamlbuild.ml /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
+ env REASON_BUILD_DIR=../../src ../../src/reopt.sh unix.cmxa -I /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa myocamlbuild.ml /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
File "myocamlbuild.ml", line 1:
Error: Files ../../src/reasonbuild.cmx
       and /Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml/ocamlbuild/ocamlbuildlib.cmxa
       make inconsistent assumptions over interface Ocamlbuild_plugin
Command exited with code 2.
Command exited with code 10.
sebmck picture sebmck on 23 Aug 2016

It may have something to do with the fact we copy over build artifacts to our cache? Do the build scripts remove any files/folders?

sebmck picture sebmck on 23 Aug 2016

The build scripts shouldn't remove anything.

One thing that looks suspicious is for every package x, we creates a symbolic link from

/Users/sebmck/Scratch/jengaboot/node_modules/x/_build/ocamlfind/lib/ocaml to
/Users/sebmck/Scratch/jengaboot/node_modules/ocaml/lib/ocaml

Maybe the cache is not handling the symbolic link correctly? Did you implement the fs diffing algorithm already?

yunxing picture yunxing on 23 Aug 2016

@kittens I fixed the issue on our end. There was a package (ocamlbuild) that depends on ocaml 4.03 while our ocaml version is only 4.02.3 (https://github.com/npm-opam/ocamlbuild/blob/master/opam).

I removed the dependency to that package and things start to make more progress.

Not sure why this issue doesn't show up in npm. But not going to dig deeper for now.

Feel free to close this issue.

yunxing picture yunxing on 25 Aug 2016

@yunxing Woo awesome!

sebmck picture sebmck on 25 Aug 2016

FYI: I'm able to install reason, jenga (our future build system) and stdlib now via kpm! :
image

yunxing picture yunxing on 25 Aug 2016

Wonder what's causing the weird progress bar output...

sebmck picture sebmck on 25 Aug 2016

@kittens oh I pressed some "ENTER"s during installation....

yunxing picture yunxing on 25 Aug 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

sebmck picture sebmck  ·  3Comments
catkins picture catkins  ·  3Comments
baptistelebail picture baptistelebail  ·  3Comments
mnpenner picture mnpenner  ·  3Comments
esphen picture esphen  ·  3Comments