Pnpm: In a multi-package repository, link dependencies from a shared node_modules

[zkochan]

As of v2.15, pnpm installs an isolated node_modules for every project in a multi-package repository. This approach has some downsides:

1. Dependencies are not singletons (see #604)
2. The same dependencies are built several times (once in every project where they are installed)
3. It takes a lot of time to create thousands of hard links in every project.
4. Every project has a huge shrinkwrap.yaml file.

A huge pro of pnpm's solution is that projects only have access to modules that are declared in their package.json. (Yarn, for instance, creates a huge flat node_modules in the root and so projects have access to lots of modules that are not declared in their package.jsons)

Solution

pnpm stores all dependencies in an isolated subfolder called node_modules/.registry.npmjs.org/ (more about this here). Every sub-dependency is stored in that subfolder but only the root dependencies are linked into node_modules.

In case of a multi-package repository, pnpm can store all the dependencies inside <repo root>/node_modules/.registry.npmjs.org/. Projects of the repo won't have any hard links inside node_modules. All their dependencies will be symlinked from the <repo root>/node_modules/.registry.npmjs.org/.

Information about the packages will be saved in a single shrinkwrap.yaml file in the root of the repository.

New shrinkwrap.yaml format (a shared shrinkwrap)

should the new shared shrinkwrap has a different file name?

When a shrinkwrap.yaml will be used by several projects, it will have a new field called projects. This field will be a map of project path to project specifiers/dependencies/devDependencies/optionalDependencies. For instance:

importers:
  components/button:
    dependencies:
      foo: 1.0.0
    specifiers:
      foo: ^1.0.0
  components/page:
    dependencies:
      button: link:../button
    specifiers:
      button: *
packages:
  /foo/1.0.0:
...

Implementation

In order to have an early functional implementation:

• supi will have the possibility to install using such "shared" lockfile
• supi will be executed not concurrently (as it is now) but installation should be as fast because a lot less fs operations will be needed
• supi will not prune the node_modules during installation in every separate project. Prunning will be called after every package was installed

Related

We are also thinking about a concept that will allow symlinking every dependency directly from the store #1001. However, that solution is a bit extreme for now and harder to implement. This may be a good compromise for the time being.

TODO

• [x] implement the shared shrinkwrap format #1373
• [x] improve the performance of installation when using a shared node_modules #1401
• [x] bump shrinkwrapVersion to 4 for the new shared shrinkwrap format
• [x] support filtering (only subdeps of the filtererd packages should be installed) bed248adc226ea6d8886c7fd9351b8d804cc585e
• [x] remove registry field from shrinkwrap version 4
• [x] make pnpm list work with the new shrinkwrap format
• [x] make pnpm outdated work with the new shrinkwrap format
• [ ] improve reporting of installation when shared shrinkwrap is used
• [ ] make headless work with several importers
• [x] support --no-link-workspace-packages

[octogonz]

This is very exciting! Some thoughts:

• Could you provide a small example of the shared shrinkwrap file format that illustrates what's new?

• What happens when a package.json is changed, but it can be solved without talking to the NPM registry? (E.g. project-a adds a dependency+SemVer that's the same as project-b in the repo.) In this situation, Rush tries to automatically fix up the shrinkwrap file without requiring this update to be tracked by Git, because the install behavior is still fully deterministic. This reduces churn/merge conflicts for the shrinkwrap file, which can be very expensive. Should PNPM have such a feature? If not, would it be possible to implement it on top of PNPM?

• Maybe the name "mono shrinkwrap" or "monorepo shrinkwrap" would be more clear? "Shared" sounds a little like sharing with an external consumer. (I don't have a strong opinion here.)

@nickpape-msft @iclanton

[zkochan]

Could you provide a small example of the shared shrinkwrap file format that illustrates what's new?

Here is what would the cyclejs multi-package repo get: https://gist.github.com/zkochan/e2d7e6e7e2a805872f8dbe152672ba1e

[zkochan]

What happens when a package.json is changed, but it can be solved without talking to the NPM registry? (E.g. project-a adds a dependency+SemVer that's the same as project-b in the repo.) In this situation, Rush tries to automatically fix up the shrinkwrap file without requiring this update to be tracked by Git, because the install behavior is still fully deterministic. This reduces churn/merge conflicts for the shrinkwrap file, which can be very expensive. Should PNPM have such a feature? If not, would it be possible to implement it on top of PNPM?

This is not something new. It was implemented in pnpm v2.14.0: https://pnpm.js.org/docs/en/workspace.html#linking-packages-inside-a-workspace

In the case that you described, the installed package will be added to package.json.
For instance, if the version of project-b in the repo is 1.0.0, then: "project-b": "^1.0.0".

To shrinkwrap.yaml only something like this will be added:
project-b: link:packages/project-b

So shrinkwrap.yaml will have minimal changes

[zkochan]

Maybe the name "mono shrinkwrap" or "monorepo shrinkwrap" would be more clear? "Shared" sounds a little like sharing with an external consumer. (I don't have a strong opinion here.)

I think it can be decided later. At the moment, I would be more interested about your opinion about the config keys that I use in the PR #1373

shrinkwrap-directory: path and shared-workspace-shrinkwrap: boolean. The second config contains the word "shared". I guess it may be changed to single-workspace-shrinkwrap. But maybe "shrinkwrap" is not good for these configs. shrinkwrap-directory may be also called something like real-node-modules because that is the directory to which deps are hard linked

[octogonz]

I noticed that your shared shrinkwrap file is still called shrinkwrap.yaml. Long term, is your plan to use a unified schema for both the single-project and the multi-project (workspace) scenario? If so that seems like a good idea.

[octogonz]

In your (example file](https://gist.github.com/zkochan/e2d7e6e7e2a805872f8dbe152672ba1e) I noticed that the new importers section refers to local file paths:

examples/advanced/animated-letters:
  . . .
examples/advanced/autocomplete-search:
  . . .
examples/advanced/bmi-nested:
  . . .

Is this technically necessary? It would mean that the PNPM shrinkwrap file would be invalidated whenever a project is moved to a different folder. In our main monorepo these folder reorganizations happen fairly regularly by design. Our own Rush configuration intentionally restricts the folder depth to exactly 2 levels (i.e. project folders always have exactly one parent called the "category folder"). Before instituting this policy, we were finding that people tended to create a folder with their team name, which isolated their team's projects in a silo. The category folder requirement pushes everyone into a common namespace, so people have to talk with other teams and agree on category names that make sense. This encouraged collaboration and reduced duplication, and overall the resulting organization is more understandable for newcomers. The NPM package registry is a flat namespace anyway.

Rush already enforces that the NPM package name is unique within a monorepo, which seems like a reasonable requirement. So you could just use the name from package.json as your key, right?

[octogonz]

Just curious, why did you start tracking the specifiers field in shrinkwrap.yaml?

[vjpr]

Why is it called importers?

[zkochan]

Because they import (symlink) packages from the shared node_modules.

[zkochan]

I noticed that your shared shrinkwrap file is still called shrinkwrap.yaml. Long term, is your plan to use a unified schema for both the single-project and the multi-project (workspace) scenario? If so that seems like a good idea.

At the moment, pnpm can recognize a "dedicated" shrinkwrap.yaml from a shared one. A shared one contains an "importers" field, a dedicated does not. We can unify them by using the "importers" field in the dedicated shrinkwrap as well. I don't know if that is a good idea or not

[zkochan]

Is this technically necessary? It would mean that the PNPM shrinkwrap file would be invalidated whenever a project is moved to a different folder.

I like everything that you described but I think from performance perspective it makes sense. pnpm knows the exact location of the workspace packages, so it can avoid redundant filesystem operations

[zkochan]

Just curious, why did you start tracking the specifiers field in shrinkwrap.yaml?

This field is compared to the specifiers in the package.json. If they match, it means that the shrinkwrap.yaml is up-to-date and there is no need to reanalyze it

[octogonz]

I like everything that you described but I think from performance perspective it makes sense. pnpm knows the exact location of the workspace packages, so it can avoid redundant filesystem operations

Could you make it so that pnpm install still succeeds even if the paths are incorrect? We'd be fine with install being slightly slower until the next update, since our shrinkwrap file will get inevitably get regenerated by another PR within 24 hours. The main point is that PRs that modify Git's shrinkwrap.yaml sometimes have to get in line behind a mutex of Git merge conflicts. In other words, if the shrinkwrap modifications can be limited to common/temp/shrinkwrap.yaml (not tracked by Git) instead of common/config/rush/shrinkwrap.yaml, it makes the PR get merged much more quickly.

[zkochan]

If you don't use the --frozen-shrinkwrap flag/config then it will succeed.

[zkochan]

This is happening! I decided to ship it only once the performance is optimized. The changes in #1401 are making pnpm recursive install at least twice faster when a single shrinkwrap.yaml is used.

[zkochan]

Published with [email protected] (dist-tag: next)

[vjpr]

Awesome work! Really keen to try this out.

[zkochan]

I have experienced some issues on a big project. Some subdeps were not linked correctly.

[zkochan]

It turns out that the issue that I experienced was the same as #1305, so now I have steps to repro that annoying error.

[zkochan]

one thing to think about. This new shrinkwrap format is not backward compatible (the shared shrinkwrap with importers field), so it might be a good idea to bump the shrinkwrapVersion of it to 4. The non-shared shrinkwrap files can stay v3

but if we do that, I would also like to remove the registry field from it. Related issue: #1353

[zkochan]

2.17.0-2 released. It has many improvements. I suggest trying it out.

[zkochan]

This repo now uses a single shrinkwrap.yaml file.

[octogonz]

Cool!

[zkochan]

Version 2.17.0-5 was published. It is an RC version. If no regressions will be found, I will publish it as latest (2.17.0) in a few days.

[zkochan]

In 2.17.4 this new config works well with all commands

[vjpr]

@zkochan How do I enable this?

[zkochan]

https://pnpm.js.org/docs/en/pnpm-recursive.html#shared-workspace-shrinkwrap