Berry: [Bug] Infinite loop / OOM when using yarn 2 pnp on a larger snowpack CRA codebase

[alubbe]

• [ ] I'd be willing to implement a fix

Describe the bug
We're trying to adopt snowpack for our CRA codebase, but we have an issue that blocks us from using it.

What's happening is that we're running into a weird issue when using yarn 2 pnp with snowpack to build our frontend - it just runs in circles (probably some infinite loop or recursion) until it hits OOM.

To Reproduce

It's triggered by simply trying to build something with a large number of dependencies:

import "@date-io/date-fns";
import "@date-io/moment";
import "@material-ui/core";
import "@material-ui/icons";
import "@material-ui/lab";
import "@material-ui/pickers";
import "@material-ui/styles";
import "@redux-saga/core";
import "@rooks/use-sessionstorage";
import "@storybook/addon-actions";
import "@storybook/addon-knobs";
import "@storybook/react";
import "array-move";
import "axios";
import "axios-mock-adapter";
import "classnames";
import "core-js";
import "create-react-class";
import "css-mediaquery";
import "d3";
import "d3-format";
import "d3-scale";
import "d3-selection";
import "d3-shape";
import "date-fns";
import "downshift";
import "enzyme";
import "enzyme-adapter-react-16";
import "enzyme-to-json";
import "eventsource";
import "file-saver";
import "formik";
import "fuzzaldrin-plus";
import "history";
import "html2canvas";
import "humanize-string";
import "i18next";
import "i18next-xhr-backend";
import "immutability-helper";
import "javascript-lp-solver";
import "javascript-time-ago";
import "jsonwebtoken";
import "jss";
import "jss-keyframes";

If you run snowpack build with yarn 1 or npm, the build works fine: https://github.com/alubbe/snowpack-yarn2/tree/a0c41b6333565858c857cce87889c014d0c06518 (yarn && yarn build)

If you run snowpack build with yarn 2, the build runs forever, then OOMs: https://github.com/alubbe/snowpack-yarn2/tree/4fec8a1171c2a05ba6f6656db56da0ea2ecf185d (yarn && yarn build)

But funnily enough, just comment out a few import lines, and suddenly it works, even with yarn 2. It works regardless of what I set max-heap-size/max-old-space-size to and regardless of what import statements I comment out (so this is not caused by a particular dependency). It seems to be purely caused by the size of the imported dependencies, which is really bizarre.

So for example, with yarn 2, if you comment out

// import "jsonwebtoken";
// import "jss";
// import "jss-keyframes";

it all works. So one might think one or more of those dependencies are causing the issue, but if you put them back in and instead comment out everything else, it also builds. It's really confusing...

I'm not sure whether it's a yarn bug or a snowpack bug, so I've also created a bug report on their tracker: https://www.pika.dev/npm/snowpack/discuss/518

Any help or pointers would be greatly appreciated!

Environment if relevant (please complete the following information):

• OS: Mac
• Node version: 14.5.0
• Yarn version: 2.1.1

Additional context

It's the last issue that blocks us from adopting yarn 2 pnp in our large corporate monorepo - it would be fantastic to resolve it :)

[alubbe]

Sorry to bump this, but we are so clueless where to even look for the cause of the issues that any kind of pointer would helpful - we'd love to adopt yarn 2 pnp for our codebase!

[merceyz]

I looked into this when you originally opened it and I think I could reproduce but I can't really remember, however I just tested this on Windows 7 with Node v12.18.1 and the build passes, it's slow but it passes. Can you still reproduce with latest node version?

[alubbe]

Strange, it worked for me, as well. So just in case, I've added a new commit that adds a few more dependencies (still less than what we use in production) and tested both 12.18.3 and 14.5.0, both of (on a mac) run for a really long time and then OOM: https://github.com/alubbe/snowpack-yarn2/commit/fef7334e50fe62d251aa1b8de96e4c9a175faa30#diff-73642d587163f2eb4d72f0fa6ebfba02

[merceyz]

Could you update it to use the latest yarn version and check that? yarn set version latest

[alubbe]

woah - but if I compile yarn 2 from sources with your recent PR, it works! (see https://github.com/alubbe/snowpack-yarn2/commit/0f0ce6dd24cc3cae730fc3940512a0779c8a7d0e)
I will try it out on our real codebase next week :)

[alubbe]

Just saw your message - how do I know how recent latest is? Is https://github.com/yarnpkg/berry/blob/master/CHANGELOG.md covering that?

[merceyz]

At the time of writing latest is master https://github.com/yarnpkg/berry/commit/bc9a7a6a5152f552bcd2cd6acab7345449803fa2

if I compile yarn 2 from sources with your recent PR, it works!

I'm curious if it's actually that PR that fixes it or not, could you try with the commit before it? (e97be13977e287dc599bd96fcb7508cb2ff6c1bc)
yarn set version from sources --branch e97be13977e287dc599bd96fcb7508cb2ff6c1bc

[merceyz]

I just tested https://github.com/alubbe/snowpack-yarn2/commit/fef7334e50fe62d251aa1b8de96e4c9a175faa30 and it passes fine for me, it finishes in 3m10.581s, also tested with https://github.com/alubbe/snowpack-yarn2/commit/0f0ce6dd24cc3cae730fc3940512a0779c8a7d0e which completes in 1m51.948s (79 seconds faster)

[alubbe]

So unfortunately it still doesn't work on our full codebase and I can also still break it by adding more dependencies (closer to what we have in our repo): https://github.com/alubbe/snowpack-yarn2/commit/3a59ab2fca511a0ab192ff9c74f1787df80b7b2e

Does this also break on your windows setup? On mac with 14.8.0 I get OOM

[alubbe]

I tried setting --max-heap-size=8000, but that also runs OOM, just a bit later.
In any case, alubbe/snowpack-yarn2@3a59ab2 works just fine with node-linker, so there's something going on somewhere

[merceyz]

It runs out of memory here as well now, i'll look into it

[merceyz]

Looked into it (and found https://github.com/yarnpkg/berry/pull/1783 but that doesn't fix this problem) and I actually get a OOM error when using the node-modules linker as well, profiler shows that decodedSourcemap from Rollup is using 1.3 GB of RAM so that's the reason it runs out, PnP just helps it get there a bit faster since it has a slight memory overhead.

So it seems snowpack is preventing the sourcemaps from getting GC'ed, @FredKSchott do you have any suggestions regarding this issue?

[FredKSchott]

Thanks for pinging me, happy to help debug this! No reason for us to be spinning up 1.3 GB of RAM, with or without yarn 2 :) (PS: our discussion board is now here: https://github.com/pikapkg/snowpack/discussions, sorry I missed your original message)

Can you confirm that you haven't set installOptions.sourceMap to true in your Snowpack config file? Can you set it to false explicitly, and check if that improves things?

The other thing worth trying, which would involve modifying the Snowpack package code a bit before you run it, would be to comment out the call to packageBundle.write(outputOptions) and see if you still hit the OOM. You could also try addingsourcemapExcludeSources: true to outputOptions. Rollup docs mention this as a way to make sourcemaps significantly smaller, which could also help.

If none of that helps, I'll spin up an issue on our repo for someone to take a deeper look on our end. As far as I know, we're not doing anything special with Rollup that would cause source maps to hang around longer than expected.

[merceyz]

Can you confirm that you haven't set installOptions.sourceMap to true in your Snowpack config file? Can you set it to false explicitly, and check if that improves things?

It's not set as far as I can tell, the repro is at https://github.com/alubbe/snowpack-yarn2 where you can run this to install with node_modules instead of PnP

yarn config set nodeLinker node-modules && yarn

The stringified object looks like this (excluding the inputs as there are more than 100 of them)

{
    "input": {  },
    "treeshake": { "moduleSideEffects": "no-external" },
    "plugins": [
        { "name": "alias" },
        { "name": "snowpack:fetch-handler" },
        { "name": "node-resolve" },
        { "name": "json" },
        { "name": "snowpack:rollup-plugin-css" },
        { "name": "commonjs" },
        { "name": "snowpack:wrap-install-targets" },
        { "name": "snowpack:rollup-plugin-stats" },
        { "name": "snowpack:rollup-plugin-node-process-polyfill" },
        { "name": "node-polyfills" },
        { "name": "commonjs" },
        { "name": "snowpack:rollup-plugin-catch-unresolved" }
    ]
}

From that JSON I noticed commonjs is declared twice and that Snowpack has a node polyfill plugin so I commented out https://github.com/alubbe/snowpack-yarn2/blob/129f88423ca753f9cecfd27385792aa160091ea8/snowpack.config.js#L14-L18 which stops the OOM and the install succeeds :tada:. So this seems to be a missconfiguration issue and not a Snowpack issue, perhaps Snowpack could detect these plugins and throw a error or filter them out. :thinking:

The other thing worth trying, which would involve modifying the Snowpack package code a bit before you run it, would be to comment out the call to packageBundle.write(outputOptions) and see if you still hit the OOM. You could also try addingsourcemapExcludeSources: true to outputOptions. Rollup docs mention this as a way to make sourcemaps significantly smaller, which could also help.

It never actually gets to writing the bundle, it gets to here https://github.com/pikapkg/snowpack/blob/b070739bfb68109d6f9cb015fd670e91b3c4b046/snowpack/src/commands/install.ts#L395 sits there for a few minutes then hits a OOM

[alubbe]

I just wanted to quickly chime in that I only get the OOM with yarn 2 pnp, I can run both npm install && npm run build as well as yarn && yarn build with the node-modules nodeLinker and everything is fine. Here's the commit that works: alubbe/snowpack-yarn2@17dbd3be5ddb10a8ce8ff29c5f64a8a38a2314f1

[alubbe]

From that JSON I noticed commonjs is declared twice and that Snowpack has a node polyfill plugin so I commented out https://github.com/alubbe/snowpack-yarn2/blob/129f88423ca753f9cecfd27385792aa160091ea8/snowpack.config.js#L14-L18 which stops the OOM and the install succeeds 🎉.

But without specifically declaring nodePolyfills as a plugin I get a lot of these errors:

[snowpack] .yarn/cache/javascript-lp-solver-npm-0.4.24-918c1fc7b5-f482b2fb59.zip/node_modules/javascript-lp-solver/src/External/lpsolve/main.js
   Module "child_process" (Node.js built-in) is not available in the browser. Run Snowpack with --polyfill-node to fix.
// and many, many more errors...

and without specifically declaring commonjs as a plugin I get a lot of these errors:

[snowpack] babel-runtime/helpers/slicedToArray?commonjs-external
   Module "babel-runtime/helpers/slicedToArray" could not be resolved by Snowpack (Is it installed?).
// and many, many more errors...

What did you mean that the install succeeds?

[merceyz]

Ah, yes, you need to run the build as yarn snowpack build --polyfill-node and add these entries to the packageExtensions

react-base16-styling@*:
  dependencies:
    "babel-runtime": '*'
component-classes@*:
  dependencies:
    indexof: '*'

[alubbe]

holy moly, it works indeed (https://github.com/alubbe/snowpack-yarn2/commit/39ced354b857e1ccb0f9b909b7cb02847e3818cd)! let me try it on our larger codebase

[alubbe]

Quick intermezzo - can these warnings be ignored or could they cause trouble? I might report them to their respective repos and submit PRs fixing it:

[snowpack] Warning: 1+ circular dependencies found via ".yarn/cache/d3-selection-npm-1.4.2-7dcf4c89d3-5b0df1824a.zip/node_modules/d3-selection/src/selection/index.js".
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification
Non-existent export 'bpfrpt_proptype_WindowScroller' is imported from .yarn/$$virtual/react-virtualized-virtual-ef00da022c/0/cache/react-virtualized-npm-9.22.2-dd176935ce-e8d886b571.zip/node_modules/react-virtualized/dist/es/WindowScroller/WindowScroller.js

[merceyz]

holy moly, it works indeed (alubbe/snowpack-yarn2@39ced35)! let me try it on our larger codebase

When I test on that commit I get a OOM error, deleting the yarn.lock and reinstalling gets rid of it, so seems a certain amount of dependencies/specific dependencies still cause problems. decodedSourcemap is still using 1.3GB RAM now so the issue isn't resolved. I'll cc @lukastaegert (hope you don't mind) in case this is a issue in rollup or if they have any ideas how to debug this further

Note: when profiling, the PnP runtime is only using a few MB of RAM so it's not that causing it.

[alubbe]

I can now report that everything is working for us - thank you all!

[larixer]

@alubbe I suspect it might be related to the PRs: #1888 and #1838

[merceyz]

@alubbe That's great, are you able to share what fixed it / what the problem was?

[alubbe]

Upgrading yarn and snowpack in tandem - both codebases improved performance/fixed bugs and now it just works 🎉