Pnpm: Frameworks/toolings don't resolve dependencies the way Node does

[zkochan]

We have a lot of issues that happen because toolings like webpack, browserify and others don't resolve dependencies the way Node does. This is something that pnpm will never fix and the toolings should fix. And they should fix it not because of pnpm but because they should be Node-compatible.

Open issues we currently have because of this:

• [ ] webpack: #740
• [x] browserify: #795
• [x] angular (because of webpack): #769
• [x] CRA (because of webpack configuration?): #581
• [ ] eslint (actually just a plugin of it but a widely used one): #739
• [x] jest: #1007
• [ ] maybe karma: #720

Help is highly appreciated here. If you have friends in those communities or are familiar with their codebase please try to make this happen faster.

Also please reference this issue to make tracking of progress easier

[donaldpipowitch]

It looks like there are problems with TypeScript, too:

We use Mocha in our TypeScript-based tests which is installed as a dependency of our "build tool" (see here). Normally this is installed at ./node_modules/@types/mocha.

In our tsconfig.json we say:

{ "compilerOptions": { "types": [ "mocha" ] } }

Now we can use describe, it and so on (global variables from Mocha) inside our tests.

But with pnpm no ./node_modules/@types/mocha is created. The typings for Mocha are installed at ./node_modules/.registry.npmjs.org/@types/mocha. TypeScript can't find them and I can't change our tsconfig.json in a way that works for yarn/npm users, too.

[zkochan]

If you use mocha in your typescript project, your project has to depend on @types/mocha as well I believe. And in that case it will be in node_modules/@types/mocha

[donaldpipowitch]

Hmm... I don't use mocha directly, but through @mercateo/ws. When @mercateo/ws updates mocha to a new major version, I probably would update @types/mocha, too. But now I don't have control over it anymore. I could say it is a peer dependency, but it really is optional. (I don't have to use TypeScript for tests.)

But it would solve my problem, yeah.

Tough question 🤔

[zkochan]

hm, I just tried CRA with the latest pnpm and it works! I don't know if something in pnpm was fixed or in CRA but like a month ago it did not work

[jbergstroem]

I'm running into the same issue with karma (just to confirm), webpack and babel (edit).

Karma:

> BABEL_ENV=testing karma start test/unit/karma.conf.js

19 06 2017 17:56:52.471:ERROR [preprocess]: Can not load "webpack", it is not registered!
  Perhaps you are missing some plugin?
19 06 2017 17:56:52.474:ERROR [preprocess]: Can not load "sourcemap", it is not registered!
  Perhaps you are missing some plugin?
/Users/jbergstroem/Work/.pnpm-store/2/registry.npmjs.org/di/0.0.1/node_modules/di/lib/injector.js:9
      throw error('No provider for "' + name + '"!');
      ^

Error: No provider for "framework:tap"! (Resolving: framework:tap)
    at error (/Users/jbergstroem/Work/.pnpm-store/2/registry.npmjs.org/di/0.0.1/node_modules/di/lib/injector.js:22:12)
    at Object.get (/Users/jbergstroem/Work/.pnpm-store/2/registry.npmjs.org/di/0.0.1/node_modules/di/lib/injector.js:9:13)
    at Injector.get (/Users/jbergstroem/Work/.pnpm-store/2/registry.npmjs.org/di/0.0.1/node_modules/di/lib/injector.js:54:19)
    at /Users/jbergstroem/Work/$work/node_modules/.registry.npmjs.org/karma/1.7.0/node_modules/karma/lib/server.js:143:20

Edit: looks like we should add babel as well?

ERROR in ./src/index.js
Module build failed: Error: Couldn't find preset "es2015" relative to directory "/Users/jbergstroem/Work/$work"
    at /Users/jbergstroem/Work/$work/node_modules/.registry.npmjs.org/babel-core/6.25.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:293:19
    at Array.map (native)
    at OptionManager.resolvePresets (/Users/jbergstroem/Work/pageload/bosque/node_modules/.registry.npmjs.org/babel-core/6.25.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
    at OptionManager.mergePresets (/Users/jbergstroem/Work/pageload/bosque/node_modules/.registry.npmjs.org/babel-core/6.25.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
    at OptionManager.mergeOptions (/Users/jbergstroem/Work/pageload/bosque/node_modules/.registry.npmjs.org/babel-core/6.25.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
    at OptionManager.init (/Users/jbergstroem/Work/pageload/bosque/node_modules/.registry.npmjs.org/babel-core/6.25.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)

[zkochan]

@jbergstroem what version of webpack? It seems like with the latest webpack (2.6.1) react apps and angular apps work fine. Although they did not work a week or so ago

[jbergstroem]

@zkochan it is indeed 2.6.1 so it might be using it with babel that breaks (classic es2015 transpile job). Just popped up on my screen so I might have been a bit to premature coming to conclusions.

[zkochan]

@jbergstroem does the workaround from this issue solve your problem? https://github.com/pnpm/pnpm/issues/720

[jbergstroem]

@zkochan not really;

    frameworks: [require('karma-tap')],

19 06 2017 19:32:52.567:ERROR [preprocess]: Can not load "[object Object]", it is not registered!
  Perhaps you are missing some plugin?
/Users/jbergstroem/Work/.pnpm-store/2/registry.npmjs.org/di/0.0.1/node_modules/di/lib/injector.js:9
      throw error('No provider for "' + name + '"!');
      ^

Error: No provider for "framework:[object Object]"! (Resolving: framework:[object Object])

Edit: I get your point; sorry -- new to karma. I'll load them as plugins and pass them as usual.

[zkochan]

@jbergstroem you have to register it via passing it to plugins. And then you can just pass the name to frameworks

frameworks: ['tap'],
plugins: [require('karma-tap')],

[jbergstroem]

@zkochan: yeah, I did -- thanks for taking the time to follow up though. I'm kind of walking through the webpack/babel/karma stack now seeing how they all seem to enjoy the same inclusion pattern.

[zkochan]

so it worked, right?

[jbergstroem]

@zkochan said:
so it worked, right?

Yes, it worked -- sorry should've clarified.

[stereokai]

I just discovered yesterday and I really believe in the idea. I would really like to help with this as obviously it's a major roadblock. Where to start?

[zkochan]

It seems to work fine with latest webpack. So the react stack and Angular2 work fine. (they did not a week or so ago).

So the eslint #739 issue and the browserify one #795 are next in priority.

[vkrol]

I also have problems with Jest.

[zkochan]

I added jest to the list. As far as I remember jest does not resolve modules to their real path, which is not how node resolves dependencies. That is why it doesn't currently work with pnpm

[zkochan]

An update on this. I did a PR to resolve to add a preserveSymlinks: false option to it. We can now create PRs to any tools that use resolve and pass the preserveSymlinks: false option to make it work with pnpm. Also, this option will be false by default in resolve@2.

[stereokai]

@zkochan All we need to do now is wait for all those tools to bump their version of node-resolve?

[zkochan]

A new feature was added to pnpm in v1.12 that allows us to hook into the installation process and substitute some of the packages. This is very handy if framework/tools are unwilling to fix their project for pnpm or the PRs are on hold for a long time period.

I wrote an article about it: Why package managers need hook systems

[zkochan]

I just created an issue at Node.js to ask for a public API of the module resolution algorithm.

https://github.com/nodejs/node/issues/16389

[NicoCevallos]

Hi, yesterday I was dealing with a similar problem and I as could see at least one of the modules that throw the error is trying to find the dependency using require('../node_modules/[dependency]') instead of require('[dependency]'), where [dependency] is the name of the required module, so require could not find the module due it is into node_modules/.registry.npmjs.org folder.
I think it could be fixed if the secondary dependencies are saved into node_modules folder as npm does.

One of the solutions I found is installing the dependency, but there are other modules (one of my rivals was jest) that has a more complex error and really I couldn't fix.

Also, latest comment/question, why pnpm doesn't use hard links into .registry.npmjs.org folder?

Thanks in advance for all the support you can give me.

[zkochan]

Packages should never require dependencies that are not declared in package.json. pnpm will not implement hacks to support buggy behavior. See pnpm’s strictness helps to avoid silly bugs.

Also, latest comment/question, why pnpm doesn't use hard links into .registry.npmjs.org folder?

pnpm uses hardlinks to import packages from the global store to the project's node_modules. Inside the project's node_modules, it uses symlinks. Otherwise, it would not work with Node's resolution algorithm. See Symlinked node_modules structure

[zkochan]

@etamponi has implemented a new flag called --shamefully-flatten which has been released with [email protected]. This flag can be used as a temporary fix to make pnpm work with toolings that rely on flat node_modules

[ishitatsuyuki]

Rollup has multiple issues:

https://github.com/rollup/rollup-plugin-node-resolve/issues/98
https://github.com/rollup/rollup-plugin-node-resolve/issues/100

[tunnckoCore]

I really don't see how one can make both ESLint and PNPM work.

Don't know why, but I am trying again with PNPM and monorepo approach. Pnpm is freaking fantastic and is totally for me, because I'm on pretty old hardware and both NPM and Yarn are against me. Yea, fetching and requesting is magically fast and this not bother me, it's near instant even with slow connections (I'm with great one, but just to say it). But the linking is awful. Both Yarn and NPM are creating some temp directories and files in the /tmp, which is kinda okey - 1) sometimes if I don't cleanup for a long time and don't restart my PC for days or weeks... it's really starting to be hundreds of folders, and I'm sure that it slow the whole machine dramatically and also eats RAM & Swap. 2) if I cleanup, e.g. rm -rf /tmp/yarn* /tmp/npm* it heals the situation for a while and linking 8-10-20k packages (8k is best cases scenario with just ESLint and your config [airbnb for example]) is okey, but for very short time.

And lets talk about ESLint (and probably other tooling that are using presets and shareable configs). It's just impossible to get it working. Looking more deeply on that, I can consider that it definitely feels like some problem with Pnpm.

Is there anyone that is using Pnpm and ESLint (both normal and monorepo scenarios)? I thought I will look what is done in this repo, but it uses TypeScript, so..

@zkochan what's the current workaround? It's not only a case with the eslint-plugin-import and its resolvers. It's no matter what eslint config you are using, it just not structures the directories correctly (i'll push a repo in a bit to show).

[zkochan]

Looking more deeply on that, I can consider that it definitely feels like some problem with Pnpm.

No, it is a problem in eslint. Here's the related issue. Seems like @ishitatsuyuki created a PR to resolve it!

W/o the fix in eslint, there are 2 workarounds:

1. Use shamefully-flatten=true
2. Install as direct dependencies the packages that eslint-plugin-import cannot find

[tunnckoCore]

Shamefully flatten doesn't help, for the last couple of hours, I played few dozen variants, with and without it.

I'll report back when I create a demo repo and explanation.

[dandv]

Worth adding this WebStorm issue with eslint-plugin-jest?

[zkochan]

Angular 8 supports pnpm
https://twitter.com/mgechev/status/1098449036351221761

[fjorgemota]

By the way I think the issue in webpack's case isn't with webpack itself, but with the package enhanced-resolve, that Webpack uses underneath to resolve the files asynchronously (according to webpack documentation), right?

Also I didn't understood how CRA and Angular resolved the problem between PNPM and Webpack. Is there any plans to create a tutorial or something? --shamefully-flatten works quite well here but it's sad to have to enter that argument everytime I want to use pnpm.

[tunnckoCore]

@fjorgemota: it's sad to have to enter that argument everytime I want to use pnpm.

Add it in .npmrc if I remember correctly. I was adviced that.

But still. Even with this flag the things not always work.

@zkochan: Angular 8 supports pnpm

What that means?! :laughing:

[fjorgemota]

oh, thanks @tunnckoCore. That seems to fix at least that problem, which is already quite good. :D

Thank you! :)

[torbensky]

Add to the list of tools Parcel

I submitted a PR on Parcel that fixes it for me.

[torbensky]

I think another PR may be needed for updating their usage of resolve. I noticed that they are not passing the preserveSymlinks flag. At first I tried fixing my problem by setting that but I believe (still very unfamiliar with their codebase) that resolve is used for things like the Parcel plugins which I haven't tried using with pnpm yet.

When I have some time I'll see about updating their usage of resolve as well. I'd like to test using a Parcel plugin in a pnpm project.

[beenotung]

Maybe also add the typescript plugin of intellij IDEA into the list.
When typescript packages are installed by pnpm, the auto import hint doesn't shows up.
When installed with npm, it can suggest import statement automatically.
(don't know where can I fire this issue to the right person)

[cdfa]

It seems eslint-plugin-import isn't the only one having dependency problems (I just encountered some with zoe. It seems eslint isn't able to resolve transitive dependencies in general, which leads to packages like eslint-config-prettier-standard having to list a ton of peer dependencies and therefore a lot of bloat on the user side.

Wouldn't it be better to fix this on eslint's side instead of having all config packages list all the other configs and plugins they depend on as peer dependencies?

[ExE-Boss]

@cdfa Well, https://github.com/eslint/rfcs/pull/14 might improve this situation in ESLint.

[zkochan]

I am tired of these issues, so I am working on a new version of the --shamefully-flatten flag. Subdeps will have access to unlisted deps but application code will not.

Issue: #1998
PR: #1997

I plan to make this feature on by default for pnpm v4

[tunnckoCore]

Woohoo! :tada: Interesting. Any ETA on when this?

[zkochan]

The code in the PR already works.
The first beta will be out soon

[tunnckoCore]

Whoaaa! Great.

So to expect monorepo with ESLnt, Babel, Jest (and others) to start working okay? :thinking:

[zkochan]

yes.
but they already work ok if you install them in the root workspace package and you use the shamefully-flatten config

[zkochan]

First beta version of v4 is out: 4.0.0-0
It can be installed via pnpm install --global pnpm@next.

[3cp]

I plan to make this feature on by default for pnpm v4

In v4, do I need hoist=* to get a flat node_modules?

[zkochan]

no. By default, all deps are hoisted. You can disable this by setting hoist=false. Or you can hoist a subset of deps by setting a pattern. For instance, hoist-pattern=eslint-*.

[3cp]

This surely will boost pnpm adoption.

[tunnckoCore]

Yeaaaah!! Definitely will try it! :)

[zkochan]

v4 is now latest. Most of these issues should be gone with the default configuration of pnpm