Yarn: Allow child packages to install into a single node_modules in the root package

[StephanBijzitter]

Do you want to request a _feature_ or report a _bug_?
I want to request a feature.

yarn install currently does the same as npm install, it installs the package in the current working directory. That's good, totally expected.

We have a repository that contains one root package, which contains a lot of developer dependencies (like Jest, ESLint, etc). Inside this repository, we have a modules directory which in turn has a new subdirectory for each package. As such, it could be visualised like this:

• package.json
• modules/
  
  
  
  • module-a/
  
  
  • package.json
  
  
  • module-b/
  
  
  • package.json
  
  

Currently, with npm, we parse the modules subdirectories for package.json files and cd to each of them to perform an npm install. In this example above, we would have three node_modules directories. A lot of these modules (a and b), share dependencies, even with the same version ranges. These are currently installed multiple times, which is unwanted.

We would like to be able to install these three package.json files into a single node_modules directory in the root of the project. This means if module-a and module-b both require jQuery, we would only need to install it once instead of twice (or twenty times...).

This should be a command line option, something like: yarn install --packages=package.json,modules/*/package.json.

It would parse all matching package.json files and create one big dependency tree, and then install this dependency tree into one node_modules directory that is at least accessible by the top-level package.json.

This would also allow a single lock file to be used, instead of (in this example), 3.

Please mention your node.js, yarn and operating system version.
Node 6.5.0, Yarn 0.15.1, Ubuntu 14.04 (VM)

[Daniel15]

One idea we had was to hardlink packages rather than copying them into node_modules (#499). That would at least solve disk space usage as there'd only be a single copy of the files. It'd still mean there's multiple node_modules directories though. Would that at least help a little bit?

[StephanBijzitter]

Yeah, that'd be a big improvement already. When you work with a lot of different projects, disk space is more limiting on a Macbook than RAM is (all those VMs... pfft).

[Daniel15]

I like this idea, this is something I'd also find useful.

all those VMs

Heh, when testing the various Yarn packages for our public release this morning, I had Debian, Ubuntu, CentOS, Windows 7 and Windows 10 VMs running concurrently... Basically every common OS except Mac OS since I don't have a Mac. Luckily I have lots of RAM on my home PC 😛

[Pauan]

I'm not sure if this helps in your use case, but have you tried using local packages?

The root package.json would look like this:

{
  "dependencies": {
    "module-a": "file:modules/module-a",
    "module-b": "file:modules/module-b"
  }
}

Note: Using file: causes it to use a local package, rather than a package from npm's registry.

Then the modules/module-a/package.json would look like this:

{
  "name": "module-a",
  "version": "1.0.0",
  "dependencies": {
    "jquery": "^3.1.1"
  }
}

And the modules/module-b/package.json would look like this:

{
  "name": "module-b",
  "version": "1.0.0",
  "dependencies": {
    "jquery": "^3.1.1"
  }
}

Now if you run yarn install in the root directory, it will recursively install all of the dependencies for module-a and module-b and will create a single node_modules folder and a single yarn.lock file, as you requested.

[StephanBijzitter]

@Pauan, it's so simple yet so ingenious.
Your solution to getting a single installation folder combined with Daniel's response, would be so awesome.

[jamiebuilds]

It'd probably be better to push this off into Lerna unless we decide to merge Lerna into Yarn which has been brought up before

[guncha]

We have the same setup as @StephanBijzitter and are using yarn --modules-folder=../node_modules to recursively install all of the dependencies in the top level node_modules.

This is our top level preinstall script:

MODULES=$(pwd)/node_modules
PACKAGES=$(find . -name package.json -not -path '*/node_modules/*' -mindepth 2)
for project in $PACKAGES; do
  (
    cd $(dirname $project)
    echo ==== $(pwd) ====
    yarn --modules-folder $MODULES
  )
done

This sort of works, but is highly unsafe. In case of conflicts, a later package will overwrite the dependencies in node_modules without a warning, for example.

What we would really like is an ability to hoist dependencies to the highest node_modules in the tree.

For example, the following three submodules have different dependencies on moment and two of them can use the same version, so it can be hoisted up a level.

• package.json dependencies: {}
• module_a/
  
  
  
  • package.json dependencies: { "moment": "^2.13.0" }
  
  
• module_b/
  
  
  
  • package.json dependencies: { "moment": "2.15.0" }
  
  
• module_c/
  
  
  
  • package.json dependencies: { "moment": "2.12.0" }
  
  
  • node_modules/
  
  
  • moment/
    
    
    
    
    
    • package.json version: "2.12.0"
    
    
    
  
  
• node_modules/
  
  
  
  • moment/
  
  
  • package.json version: "2.15.0"
  
  

While the module_c that requires a different version will still get that version because it will be installed in its node_modules. It's also important that any dependencies for module_a and module_b that depend on their own version of moment do not have it installed in module_a/node_modules or module_b/node_modules.

[Pauan]

@guncha Have you tried the solution that I mentioned in this issue?

[guncha]

@Pauan I did and it would work perfectly if Yarn would symlink the file: blah dependencies into node_modules instead of copying them.

1. Because of the copying, we can't work on one of the dependencies and have the changes be reflected in the code that requires it.
2. Yarn seems to cache the file:./blah dependencies such that if you update dependencies in ./blah/package.json, it will not be taken into account when you run yarn again.

The dependency hoisting does work as expected, so that's a 👍

[Pauan]

@guncha Issue 1 is tracked in #499 and #1761 (and possibly #1334 as well).

Keep in mind that a lot of people don't _want_ the implicit sharing, because it means that if you change a library, it will affect _every single Node.js package on your computer, including unrelated applications and libraries_. That isn't a problem with npm or Yarn's current behavior. If you want to modify a package, I recommend using local dependencies, rather than npm dependencies.

I just tried your specific example with moment, and unfortunately Yarn creates three versions of moment, even though it technically could create two. I believe issue #422 is related to that. If you change module_b to use version ^2.15.0 then it does correctly dedupe the folders.

There are a few solutions:

1) Use peerDependencies to force a single version

2) Wait for issue #422 to be fixed (but maybe that issue only applies to peerDependencies?)

3) Change module_b/package.json to use version ^2.15.0

After choosing one of the above solutions, you should get the desired sharing: module_a and module_b would use the same node_modules/moment folder, whereas module_c would use a different node_modules/moment folder.

---

I don't see any bug reports for issue 2, maybe you could make a new bug report? As a workaround, does running yarn upgrade work?

[heikomat]

Where i work, we use the same project structure, with local modules, that themselves can have local modules etc. We too faced the problem of unnecessary duplicate dependency installs (which made the install bigger and slower)

To solve this, i created a postinstall script, that installs all the dependencies of local modules and submodules etc into the root-node_modules, except when conflicting dependencies are found, those are installed into the associated moudle-node_modules.
It also symlinks all the local modules to the root-node_modules, so they can be required without navigating.

When looking for options to speed up installation time even more, i came across this issue. I tested my script with yarn, and although my script internally uses npm, running yarn in a project instead of npm install worked just fine.

The package ist called minstall. If you are interested in how exactly this script installs the modules, you can look here

[bestander]

We are addressing this issue in https://github.com/yarnpkg/rfcs/pull/60

[bestander]

Implemented in workspaces