Webpacker: Yarn 2.0 (Yarn Plug’n’Play)

[jakeNiemiec]

Summary from whitepaper:
Pros:

• smaller install
• faster install
• better matches how gems behave
• better install & cache for CIs
• better peer dependencies
• users working on multiple projects across a system won’t pay in-creasing install costs
• no node_modules folder

Cons:

• no node_modules folder (some webpacker stuff currently relies on this). You also can't edit anything in node_modules
• packages must be accessed via yarn cache
• packages cannot rely on cross package boundaries (e.g.: webpacker users can't rely on the sub-dependencies of @rails/webpacker as if they were direct dependencies)

Resources:

• overview
• getting-started
• troubleshooting
• whitepaper
• webpack plugin

Newer links:

• https://dev.to/arcanis/introducing-yarn-2-4eh1
• https://next.yarnpkg.com/getting-started
• https://next.yarnpkg.com/advanced/migration

[omnilord]

Will pnp put all the modules in a singular global source of truth location on the local system, or will this still require downloading at least one new copy of them from the internet at every install, even if another project already uses the same version?

[jakeNiemiec]

From the whitepaper (very short read):

In its current state, running yarn install does the following under the hood -assuming a cold setup:

1. Dependency ranges are resolved into pinned versions
2. Packages for each version are fetched from their sources and stored in the offline mirror
3. The offline mirror is unpacked into the cache
4. The cache is copied into thenode_modulesfolders

Our solution aims to trim the fourth step from the equation. Instead of copying each and every file from the cache to the node_modules folder (which can take a lot of time, amplified by the sheer number of files), we will instead generate a single file that will contain static resolutions tables that list:

• What packages are available in the dependency tree
• How they are linked together
• Where they are located on the disk

So, instead of 100M of modules in a node_modules folder, that 100M is cached and you get a file that acts as a map to the modules that you need. No more filesystem traversal.

[paracycle]

PnP was exactly what I thought about when I read the thread at https://github.com/rails/rails/pull/36282, great to see there is a will to tackle this.

I would be interested in taking a look at implementing this but I am _super_ new to webpacker. Let me see how fast I can get familiar with it to make a contribution.

[jakeNiemiec]

I have tried it out and can confirm that webpacker needs to implement something like peerDependencies in order to make this work. Error with the default setup: You cannot require a package ("@babel/preset-env") that is not declared in your dependencies:

Add this to package.json to enable.

"installConfig": {
    "pnp": true
  }

The errors

Uncaught Error: Module build failed (from /Users/user/Library/Caches/Yarn/v4/npm-babel-loader-8.0.6-e33bdb6f362b03f4bb141a0c21ab87c501b70dfb/node_modules/babel-loader/lib/index.js):
Error: You cannot require a package ("@babel/preset-env") that is not declared in your dependencies (via "/Users/user/webpacker_tests/hello/babel.config.js")
    at makeError (Users/user/webpacker_tests/hello/.pnp.js:55)
    at Object.resolveToUnqualified (Users/user/webpacker_tests/hello/.pnp.js:11126)
    at Object.resolveRequest (Users/user/webpacker_tests/hello/.pnp.js:11204)
    at Function.Module._resolveFilename (Users/user/webpacker_tests/hello/.pnp.js:11386)
    at Function.Module._load (Users/user/webpacker_tests/hello/.pnp.js:11302)
    at Module.require (internal/modules/cjs/loader.js:637)
    ...

If you meticulously install all of the deps you need with PnP, the starter files will actually compile successfully:

@gauravtiwari, This is only possible since you already did some work last year to enable PnP 🎉: https://github.com/rails/webpacker/pull/1823/files#diff-1701de23f718ce36149c9b23e2db08caR95

First glance action items:

• [ ] Fix check_yarn_integrity to do something like this PR
• [ ] Find a way to hoist deps to the users "top" package.json. Here is an example that of another CLA tool doing this
  
  
  
  • [ ] Figure out which deps need to get hoisted from package.json. This would include everything require()'d in package/environments/base.js
  
  
• [x] Should PnP be the default? No, but there should be an option to enable.
• [ ] require("channels") needs to be changed to require("../channels"). Otherwise it throws an error because it looks for a package called channels.

[gauravtiwari]

Thanks @jakeNiemiec Sorry about the delay. I will take a look tomorrow and get back to you.

[somebody32]

just my 2 cents on Should PnP be the default?: https://twitter.com/sebmck/status/1137059802838757377?s=20. Obviously, Facebook's scale is a different story, but anyway

[jakeNiemiec]

Yarn 2.0 summary post

With yarn@2 released, I assume that other users will attempt to upgrade and use with webpacker. This will not work since yarn now enforces package boundaries.

[swrobel]

FWIW I attempted to use Yarn 2 about 4 months ago to see what would happen w/ Webpacker, and this is the discussion it led to on the Yarn 2 (aka berry) repo: https://github.com/yarnpkg/berry/issues/494

[jakeNiemiec]

I wanted to add this resource for anyone looking to try this out: https://next.yarnpkg.com/getting-started

It now occurs to me that migrating to Yarn 2.0 will require 2 major efforts:

• Migrating the Webpacker repo to Yarn 2
• Migrating user's Webpacker setups to Yarn 2

Both of these involve cutting reliance on node_modules. See more in the migration guide: https://next.yarnpkg.com/advanced/migration

[doits]

Shouldn't it work with nodeLinker: node-modules in .yarnrc.yml (https://yarnpkg.com/configuration/yarnrc#nodeLinker)? It doesn't use pnp then and the good old node_modules folder is still present.

[arcanis]

Yes, that would be the ideal migration for Webpacker / Rails. That's what we typically recommend in our migration guide:

Don't worry if your project isn't quite ready for Plug'n'Play just yet! This guide will let you migrate without losing your node_modules folder, and only in a later optional section we will cover how to enable PnP support (which is recommended, but not mandatory). Baby steps! 😉

[doits]

I just tested it and it seems to work with two additional steps. I did this to upgrade and it is working in my development setup:

$ yarn set version berry
$ echo "nodeLinker: node-modules" >> .yarnrc.yml
$ yarn
$ yarn add webpack@^4.0 webpack-cli@^4.0 # otherwise it will complain about not finding the binary `webpack`

Compiling in development mode!

Tested with current master version of webpacker.

[doits]

I created #2799 with the changes needed to experimentally support Yarn 2, e.g. not to break when Yarn 2 is used.

[n-rodriguez]

I created #2799 with the changes needed to experimentally support Yarn 2, e.g. not to break when Yarn 2 is used.

Thank you! It works here :)