Pnpm: Flat tree if npm 3 engine

[aecz]

This is a feature request.
Some projects only work with npm 3 due to the need of a flat tree.
My suggestion is to check the npm engine in package.json and if >=3, generate a flat tree and/or add a flag --flat-tree to the command line.

EDIT:

git clone https://github.com/zkochan/pnpm-issue394.git
cd pnpm-issue394
pnpm i
node_modules/.bin/ng build

394 has to be solved first. Then you will have this error:

 94% asset optimizationError in bail mode: [default] C:\Users\aecz\pnpm-issue394\src\app\app.component.ts:5:12
Cannot find name 'require'.

Package @typings/node is expected at the top level of node_modules but is in fact in protractor submodule.

[iamstarkov]

Can you show such projects? Also I will be glad if someone will explain me need in flat tree

[zkochan]

To be honest, I think it is bad to design an application/package in a way that it will rely on the flat structure created by npm3

However, seems like some people do rely on it. We had #96 and #346

[aecz]

For those who didnt try/check the code, the example is the default project that Angular CLI generates. This project: https://www.npmjs.com/package/angular-cli has 200K downloads in the last month. They dont seem to be willing to fix this issue. Instead they are just asking you to use npm 3 :(

[doxiaodong]

@types in typescript 2.x with dependencies such as @types/angular-ui-bootstrap will throw duplicate error.

[zkochan]

We can add an option to make pnpm create a flat tree. I was thinking about an option called flat but yarn seems to have the same option and it has a different meaning. So maybe we can use an option called --flat-tree to avoid confusion

Edit: that was suggested in the first message in the thread 😆

[zkochan]

Furthermore we can have a more sophisticated solution as well, which will not need the flag. For instance, if a package needs a dependency from the flat tree, it can require it as inherit

"dependencies": {
  "watchman": "inherit"
}

[aecz]

inherit property looks nice but it makes us dependent of pnpm and we lose compatibility with npm, or am i missing something ?
Whats the estimated effort for adding --flat-tree option ? I could help with testing, i have a various projects that fail with pnpm and seem to be due to non flat tree.

[zkochan]

oh right, if we put the inherit property into dependencies, it will cause problems with npm. But if we make a separate property, it will not do any harm to npm. Like

"inherit": ["watchmen", "less"]

but this is just an additional solution, the property and/or engine check has to be supported for sure.

I don't know what is the estimate on this because I did not research how the flat tree works yet. But this is next in my todo list after completing caching, which has an open pull request already and after closing a small issue about making global store usage the default behavior.

[zkochan]

This will work only on Node.js >= 6.3.0 because it has the --preserve-symlinks option. For older versions of Node we would have to copy the packages.

[zkochan]

I have to make a different store for flat installations.

There will be two subfolders in store: nested/ and flat/. The flat store will be used for flat installations

[aecz]

Actually I just saw that #46 emulates flat tree so the issue example should work. Maybe there is an issue with scoped packages.

[zkochan]

The issue is on Node >= v6.3.0 because we preserve symlinks there. Hence, the hack from #46 is ignored.

[zkochan]

Also #46 flattens only peerDependencies, so it doesn't achieve a real npm3 flatness

[zkochan]

@aecz landed in v0.42.0, I installed it under the next tag for now. Use the --flat-tree argument to have flat dependencies structure

[zkochan]

This might be a temporal fix before this is done #433

[aecz]

I just tested the flat tree and the 'require' issue is solved but another appeared :(
Please note that i have angular-cli installed globally with npm. Using pnpm didnt work for me, see #418.
Im also using a local store for pnpm not a global one.

npm i -g pnpm@next
npm i -g angular-cli
git clone https://github.com/zkochan/pnpm-issue394.git
cd pnpm-issue394
pnpm i --flat-tree
ng build

Issue is now:

! ENOENT: no such file or directory, unlink 'C:\Users\aecz\pnpm-issue394\~\.pnpm\.store\flat\[email protected]\_\node_modules\wrappy'

The structure of node_modules has flattened but it seems to be a concurrency problem because the file (it is actually a link) exists.

C:\Users\aecz\pnpm-issue394\~\.pnpm\.store\flat\[email protected]\_\node_modules>dir

19/10/2016  20:27    <DIR>          .
19/10/2016  20:27    <DIR>          ..
19/10/2016  20:27    <JUNCTION>     wrappy [C:\Users\aecz\pnpm-issue394\~\.pnpm\.store\flat\[email protected]\_]

Also, i noticed that instead of keeping the modules in node_modules/.store like before it is now ~\.pnpm\.store\flat. Is it the new folder structure ?

Now if i delete the node_modules folder (the flat store is kept because it is now at the root of the project) and rerun pnpm i --flat-tree, the issue above issue is gone but ng build fails with this error:

Error: Cannot find module 'ms'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\aecz\pnpm-issue394\~\.pnpm\.store\flat\[email protected]\_\debug.js:14:20)

Looking at node_modules, it is flat but now only to the first level it seems. Not all packages have moved to the top like before. We may have two different issues here.
Sorry for the long comment. Hope that helps.

[zkochan]

Also, i noticed that instead of keeping the modules in node_modules/.store like before it is now ~.pnpm.store\flat. Is it the new folder structure ?

It was a bug on windows. I published a fix with v0.42.1

[zkochan]

@aecz don't use the global angular-cli installed by npm because it won't run with the --preserve-symlinks option. You have two options: either install it with pnpm or use the one installed with the package

you can add a scripts task for that:

"scripts": {
  "build": "ng build"
}

[aecz]

I have corrected a few things on my part but still get the same wrappy error.

npm i -g pnpm@next (now says 0.42.1)
git clone https://github.com/zkochan/pnpm-issue394.git
cd pnpm-issue394
pnpm i --flat-tree

pnpm cannot finish the install (.bin folder is not created) and fails with the error:

 ! ENOENT: no such file or directory, unlink 'C:\Users\aecz\.pnpm\.store\flat\[email protected]\_\node_modules\wrappy'

So the option --flat-tree seems to use a global store now. Local store in node_modules is gone.

[zkochan]

Maybe it is something specific to Windows. I'll check later. On linux installation finishes successfully

[zkochan]

I was able to reproduce this on Windows. I'll try to fix it

[zkochan]

@aecz I published a fix under the next dist-tag. The installation will now successfully complete.

[aecz]

Install successful but im having an error with ng build now. I have deleted ~\.pnpm folder before running the test, just in case.

npm un -g pnpm
npm i -g pnpm@next (now 0.42.4)
git clone https://github.com/zkochan/pnpm-issue394.git
cd pnpm-issue394
pnpm i --flat-tree
node_modules\.bin\ng build

All fine, except last line that gives this error:

Cannot read property 'AssetUrl' of undefined
TypeError: Cannot read property 'AssetUrl' of undefined
    at Object.<anonymous> (C:\Users\aecz\pnpm-issue394\node_modules\angular-cli\node_modules\@angular\compiler-cli\src\private_import_compiler.js:10:51)

@angular/compiler-cli needs @angular/compiler 2.0.0 but is getting 2.1.1 :(
Also @angular/compiler-cli 0.6.4 is found in two different places node_modules and angular-cli/node_modules. I think it should only be found in node_modules.

I have compared the flat structure of pnpm 0.42.4 with npm 3.10.9.
Here are the differences:

• 983 items in node_modules for npm vs 987 for pnpm => 4 extra items:
  
  
  
  • .modules.yaml
  
  
  • esprima-fb
  
  
  • fsevents
  
  
  • webdriver-manager
  
  
• 13 items in node_modules/angular-cli/node_modules for npm vs 950 for pnpm

[zkochan]

@angular/compiler-cli needs @angular/compiler 2.0.0 but is getting 2.1.1 :(

but in the dependencies of @angular/compiler-cli there is "@angular/compiler": "^2.0.0". 2.1.1 should be fine

[zkochan]

ok, I see that @angular/compiler-cli 0.6.4 has a peerDependency for "@angular/compiler": "2.0.2" and it is getting 2.1.1 as you noted.

I'll try to fix this in scope of https://github.com/rstacruz/pnpm/issues/433

[zkochan]

I fixed it on branch peers, but now I have this new issue

Can it be connected with webpack?

> [email protected] build /home/zkochan/src/test/issue394
> ng build

Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
7191ms building modules                                                                  
85ms sealing
8ms optimizing 
1ms basic module optimization 
111ms module optimization
1ms advanced module optimization 
7ms basic chunk optimization        
0ms chunk optimization 
0ms advanced chunk optimization 
0ms module and chunk tree optimization 
135ms module reviving
9ms module order optimization 
12ms module id optimization
4ms chunk reviving 
0ms chunk order optimization 
24ms chunk id optimization
68ms hashing
0ms module assets processing 
159ms chunk assets processing
3ms additional chunk assets processing 
0ms recording 
0ms additional asset processing 
1855ms chunk asset optimization
 94% asset optimizationError in bail mode: [default] /home/zkochan/src/test/issue394/node_modules/.store/@[email protected]/_/src/private_import_core.d.ts:12:55 
Property 'VIEW_ENCAPSULATION_VALUES' does not exist on type '{ isDefaultChangeDetectionStrategy: (changeDetectionStrategy: ChangeDetectionStrategy) => boolean...'.

npm ERR! Linux 4.4.0-43-generic
npm ERR! argv "/home/zkochan/.nvm/versions/node/v6.9.0/bin/node" "/home/zkochan/.nvm/versions/node/v6.9.0/bin/npm" "run" "build"
npm ERR! node v6.9.0
npm ERR! npm  v2.15.11
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: `ng build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script 'ng build'.
npm ERR! This is most likely a problem with the issue394 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     ng build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs issue394
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! 
npm ERR!     npm owner ls issue394
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/zkochan/src/test/issue394/npm-debug.log

[aecz]

Is there a dist-tag for peers branchso i can test it more easily ?
I still think the flat algorithm needs some fixing because it copies @angular/compiler-cli 0.6.4 in two different places instead of one like npm does.
Also with --preserve-symlink, we shouldnt do anything about peerDependencies except showing a warning if it is missing. I believe npm does only that so we should stay compatible with it.
I think that if the structure differences i mentioned in the previous comment are fixed, ng build should work and this issue could be closed.
Another minor point is that empty node_modules folders are created in some sub dependencies. Tell me if its worth a new issue.
Im also starting to see that if a global store is used, some edge cases will require to copy (not link) some dependencies.
For example, [email protected] in .pnpm/.store/flat has 950 links in mode_modules. I would expect it to have none as the structure is flat and those links can change if a new version of a dependency is released.

[zkochan]

you mentioned this before:

13 items in node_modules/angular-cli/node_modules for npm vs 950 for pnpm

I have implemented the flat tree structure a little bit differently than in npm3 to make it work in shared stores. I had to reference all subdeps on all level, so that each dependencies can be reused in other packages.

But this difference does not change how node resolves the dependencies.

I published the peers solution under my scope @zkochan/pnpm

The latest issue might be related to webpack. I am going to try this solution

[aecz]

Unfortunately, i get an error with @zkochan/pnpm:

npm i -g @zkochan/pnpm (installs 0.42.1)
pnpm i --flat-tree

module.js:457
    throw err;
    ^

Error: Cannot find module 'eventemitter2'
    at Function.Module._resolveFilename (module.js:455:15)
    at Function.Module._load (module.js:403:25)
    at Module.require (module.js:483:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\aecz\AppData\Roaming\npm\node_modules\@zkochan\pnpm\lib\node_modules\@zkochan\logger\index.js:2:22)

[zkochan]

Sorry for that, seems like I published it badly. I'll publish to the main package once #446 gets merged

[zkochan]

During installation of the example package I also noticed that pnpm installs some packages that npm skips because they don't satisfy the engine requirements. Create an issue for it: #448

[zkochan]

We can't imitate a flat dependency tree because pnpm doesn't use preserve-symlinks anymore and we would need to copy packages like npm or yarn.

[kenberkeley]

https://github.com/egoist/poi/issues/473