Flow: Investigate if pnpm should be used instead of npm

[Artur-]

Using the Business App Starter and the package.json files mvn install produce I see the following

Reinstalling node_modules using npm in this project takes

$ rm -rf node_modules && time npm i
real    0m12.083s

and running npm install without removing node_modules:

$ time npm i
real    0m5.214s

With pnpm the numbers are

$ rm -rf node_modules && time pnpm i
real    0m5.820s

and without removing node_modules:

$ time pnpm i
real    0m1.429s

I would assume the difference is much larger with a slow network.

Additionally the size of node_modules with pnpm is 82MB
while the size when using npm is 173MB

Package files for reference

$ cat package.json 
{
  "name": "no-name",
  "license": "UNLICENSED",
  "dependencies": {
    "@polymer/polymer": "3.2.0",
    "@webcomponents/webcomponentsjs": "^2.2.10",
    "@vaadin/flow-deps": "./target/frontend"
  },
  "devDependencies": {
    "webpack": "4.30.0",
    "webpack-cli": "3.3.0",
    "webpack-dev-server": "3.3.0",
    "webpack-babel-multi-target-plugin": "2.1.0",
    "copy-webpack-plugin": "5.0.3",
    "webpack-merge": "4.2.1",
    "raw-loader": "3.0.0"
  }
}

$ cat target/frontend/package.json 
{
  "name": "@vaadin/flow-deps",
  "version": "1.0.0",
  "license": "UNLICENSED",
  "dependencies": {
    "@vaadin/vaadin-crud": "1.0.5",
    "@vaadin/vaadin-icons": "4.3.1",
    "@vaadin/vaadin-grid": "5.4.6",
    "@vaadin/vaadin-split-layout": "4.1.1",
    "@vaadin/vaadin-combo-box": "5.0.6",
    "@vaadin/vaadin-cookie-consent": "1.1.1",
    "@vaadin/vaadin-core-shrinkwrap": "14.0.0-rc7",
    "@vaadin/vaadin-upload": "4.2.2",
    "@vaadin/vaadin-dialog": "2.2.1",
    "@vaadin/vaadin-select": "2.1.5",
    "@vaadin/vaadin-app-layout": "2.0.2",
    "@vaadin/vaadin-item": "2.1.0",
    "@vaadin/vaadin-board": "2.1.0",
    "@vaadin/vaadin-charts": "6.2.3",
    "@vaadin/vaadin-notification": "1.4.0",
    "@vaadin/vaadin-grid-pro": "2.0.3",
    "@vaadin/vaadin-progress-bar": "1.1.2",
    "@vaadin/vaadin-shrinkwrap": "14.0.0-rc7",
    "@vaadin/vaadin-ordered-layout": "1.1.0",
    "@vaadin/vaadin-login": "1.0.1",
    "@vaadin/vaadin-button": "2.2.1",
    "@vaadin/vaadin-date-picker": "4.0.3",
    "@vaadin/vaadin-text-field": "2.4.8",
    "@vaadin/vaadin-menu-bar": "1.0.3",
    "@vaadin/vaadin-custom-field": "1.0.6",
    "@vaadin/vaadin-form-layout": "2.1.4",
    "@vaadin/vaadin-confirm-dialog": "1.1.4",
    "@vaadin/vaadin-accordion": "1.0.1",
    "@polymer/iron-list": "3.0.2",
    "@vaadin/vaadin-list-box": "1.1.1",
    "@vaadin/vaadin-details": "1.0.1",
    "@vaadin/vaadin-checkbox": "2.2.10",
    "@polymer/iron-icon": "3.0.1",
    "@vaadin/vaadin-time-picker": "2.0.2",
    "@vaadin/vaadin-context-menu": "4.3.12",
    "@vaadin/vaadin-tabs": "3.0.4",
    "@vaadin/vaadin-radio-button": "1.2.3",
    "@vaadin/vaadin-lumo-styles": "1.5.0",
    "@vaadin/vaadin-material-styles": "1.2.3",
    "@vaadin/vaadin-rich-text-editor": "1.0.4"
  }
}

[Artur-]

Numbers for yarn

$ rm -rf node_modules && time yarn install
real    0m7.496s

$ time yarn install
real    0m0.632s

[Legioth]

A small note is that the very initial use of pnpm is not as much faster when its own cache is not populated. For that case, npm and yarn each takes around 30 seconds and pnpm is around 20 seconds on my machine. Subsequent runs once the global caches are populated are in line with the originally reported numbers.

[web-padawan]

Some benefits of using yarn over npm:

1. yarn does dedupe on install which would help to prevent issues like #6469
2. yarn.lock is generally more stable than package-lock.json, and platform independent
3. yarn installs dependencies a little bit faster than npm
4. yarn workspaces would make @vaadin/flow-deps work better (prototype)

[denis-anisimov]

yarn does dedupe on install which would help to prevent issues like #6469

This is really an important feature.
Other items don't look as significant improvements comparing to pnpm (see my comments about it below).

[denis-anisimov]

My numbers and comments for pnpm .

I'm using bakery app.

npm install numbers:

• (without node_modules): 0m12.479s
• (with already installed node_modules): 0m6.502s

pnpm install numbers:

• (the very first pnpm run without node_modules) 0m29.568s (and this is significantly worse than npm)
• (with already installed node_modules): 0m1.750s

• (without node_modules, second run): 0m6.293s
• (with already installed node_modules): 0m1.910s

The numbers are quite stable.

So the very first pnpm run is worse than npm.
But once it's done initially it works faster.

pnpm uses ~/.pnpm-store/ folder (like ~/.m2 ) to cache node modules and links inside node_modules to the modules inside ~/.pnpm-store/ so it doesn't redownload them across the projects which speeds up everything dramatically.

pnpm is installed as /usr/local/bin/pnpm which is a link to /usr/local/lib/node_modules/pnpm/bin/pnpm.js.

[denis-anisimov]

pnpm uses pnpm-lock.yaml instead of package-lock.json and its not in JSON format.
It looks like that:

dependencies:
  '@polymer/polymer': 3.2.0
  '@vaadin/flow-deps': 'link:target/frontend'
  '@vaadin/vaadin-combo-box': 5.0.6
  '@vaadin/vaadin-date-picker': 4.0.3
  '@webcomponents/webcomponentsjs': 2.2.10
  vaadin-combo-box: 3.0.0-alpha4
  vaadin-date-picker: 2.0.4
devDependencies:
  copy-webpack-plugin: [email protected]
  raw-loader: [email protected]
  webpack: [email protected]
  webpack-babel-multi-target-plugin: 2.1.0_9250801cdd9dbb2a6b2c5c26f1004333
  webpack-cli: [email protected]
  webpack-dev-server: [email protected]
  webpack-merge: 4.2.1
lockfileVersion: 5.1
packages:
  /@babel/cli/7.6.0_@[email protected]:
    dependencies:
      '@babel/core': 7.6.0

[caalador]

What about installation?

Can it be automated as many of our users seem to not want to install any frontend tools and having suddenly to install 3 node, npm, pnpm will drive them over the edge.

On one hand having prepare-frontend run npm install for the tools would make it possible to install pnpm locally and then for component build use pnpm. On the other it would leave us with a slow npm i for prepare-frontend

[denis-anisimov]

Can it be automated as many of our users seem to not want to install any frontend tools and having suddenly to install 3 node, npm, pnpm will drive them over the edge.

I'm not aware of any maven plugin for this at least.
But pnpm is installed via npm: npm add -g pnpm.
But that installs it globally.

It can be installed locally via npm i pnpm.
It will be installed as a JS script: node_modules/pnpm/bin/pnpm.js.

So we may install it automatically assuming npm is installed.

[platosha]

There is a potentially major difference between pnpm and both npm / yarn to note, and it is a lack of deep dependencies flattening, aka node_modules directory strictness: https://www.kochan.io/nodejs/pnpms-strictness-helps-to-avoid-silly-bugs.html

With pnpm, any custom Polymer templates (ES modules) in the ${project_dir}/frontend directory would likely:

• not be able to import dependencies installed as a result of Flow @NpmPackage dependency annotation,
• and not be able to import any transitive dependencies from meta packages like @vaadin/vaadin or @vaadin/vaadin-core,

... unless the root-level package.json explicitly specifies the imported dependencies.

For example, let’s assume we have Flow app with a @NpmPackage("@vaadin/vaadin-button") dependency annotation. This produces in a dependency entry in ${project_dir}/target/frontend/package.json, but not in project root-level ${project_dir}/package.json as of now.

With this setup, running install with both npm and yarn would create ${project_dir}/node_modules/@vaadin/vaadin-button. But, with pnpm, the dependency is installed only into ${project_dir}/target/frontend/node_modules/@vaadin/vaadin-button, which makes it available to Flow imports but unavailable to the ${project_dir}/frontend code.

Looks like pnpm has a config setting for the flat-style install called shamefully-hoist. But, as of today, it does not help me in case of flattening the ./target/frontend dependencies onto the root level, probably because ./target/frontend itself is not a regular package, but a symlink root-level dependency.

Also note that this install strictness makes meta-packages (@vaadin/vaadin, @vaadin/vaadin-core, and such) useless, as depending on them does not automatically make their dependent packages available for the user’s code.

[heruan]

@platosha Why the need of the target/frontend/package.json instead of having a single package.json in the root directory? This could also be related to #6739 (about the location/need of target/frontend).

[caalador]

The other package.json is there so that we may update it without needing some sort of this is ours and this is the users manual additions feature.
With the current setup the user can manage the main package.json as they want (except for some of the Flow dev dependencies).

[platosha]

BTW by design, Node does resolve dependencies from parent packages. Thus, listing auto-generated dependencies in root-level package.json would technically make them available for the nested target/frontend package too, regardless on whether the strict pnpm or a traditional npm/yarn install style is used.

[heruan]

Thank you @caalador for the explanation. Let me express my humble opinion: after a couple of months working with V14+NPM I can say I'd prefer having a single package.json, updated by the plugin when needed, to know which modules I'm actually depending on and to keep them tracked on version control.

[pleku]

Closing this as the investigation has concluded and pnpm support will be WIP soon (#6966)