Pnpm: The global store should always be on the same disk on which installation is done

[zkochan]

pnpm version:

0.64.8

Code to reproduce the issue:

Run pnpm install no a secondary disk (lets say it is disk D), not the one on which your system lives.

Expected behavior:

Installation is successful and a store on disk D is created/used

Actual behavior:

A store on disk C is used and pnpm fails because hardlinks cannot be cross-disk.

Additional information:

• node -v prints: any
• Windows, OS X, or Linux?: Windows

Temporary fix

If you want to do installations on a different disk till this is not fixed, just change the location of your global store to a directory on the disk you wanna use.

npm config set store-path <path to the store>

[ghost]

in my case MacOS (X), i'll try it out on Ubuntu later (might work, not sure)

ERROR EXDEV: cross-device link not permitted,
link '/Users/dym/.pnpm-store/1/registry.npmjs.org/gm/1.23.0/.npmignore'
-> '/Volumes/Data/work/site/node_modules/.registry.npmjs.org/gm/1.23.0/node_modules/gm+stage/.npmignore'

[zkochan]

Seems like this package can help us implement this https://www.npmjs.com/package/drivelist

[pstoev]

@zkochan how do you think that check would work? Checking against only physical drives? I think the issue is reproducible on logical drives as well...

[zkochan]

Yes. You are right. This seems easy to fix on Windows.

I am not so sure how to distinguish different partitions/disks on Linux and
MacOS

On Mon, Apr 24, 2017, 13:37 Plamen Stoev [e-mail] wrote:

@zkochan https://github.com/zkochan how do you think that check would
work? Checking against only physical drives? I think the issue is
reproducible on logical drives as well...

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/pnpm/pnpm/issues/712#issuecomment-296613475, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1pm3VYFBBQcCucslKzVAbuL0Q2c73Zks5rzHuAgaJpZM4NEsZs
.

[zkochan]

@noformnocontent could you please check on your environment whether path.parse correctly detects the root on your different disks?

[ghost]

tried on Ubuntu,

ERROR EXDEV: cross-device link not permitted,
link '/home/dym/.pnpm-store/1/registry.npmjs.org/morgan/1.8.1/HISTORY.md' ->
'/media/dym/Data/work/site/node_modules/.registry.npmjs.org/morgan/1.8.1/node_modules/morgan+stage/HISTORY.md'  

path.parse:

> path.parse('/media/dym/Data/work/site/package.json')
{ root: '/',
  dir: '/media/dym/Data/work/site',
  base: 'package.json',
  ext: '.json',
  name: 'package' }

[zkochan]

thanks, I hoped it would return /media/dym as root. In this case we'll have to use the drivelist package. I haven't found anything else

[ghost]

pattern is /media/<user>/<mounted_volume>, so in that case /media/dym/Data
i'll try the path.parse on mac later today

what happens if you run pnpm in a Docker/Kubernetes enviroment -- should be no problem, right?

[zkochan]

I guess if it has one filesystem then it should be fine

[zkochan]

OK, so on Windows it will be <disk letter>:\

on Linux /media/<user>/<mounted_volume>

on OSX /Volumes/<mounted_volume>

Seems easy.

[minecrawler]

on Linux /media//

what if I do

$ sudo mount /somedir && cd /somedir && pnpm install

on Linux, with /somedir defined in /etc/fstab? What about

$ sudo mount /dev/sdc1 /mnt/tmp && cd /mnt/tmp && pnpm install

?

On Linux, any directory might be on a different disk! Not everyone uses Ubuntu and its way of handling some things...

[pstoev]

thanks, I hoped it would return /media/dym as root. In this case we'll have to use the drivelist package. I haven't found anything else

@zkochan wouldn't it be enough if we "extract" (e.g. dir.substring second slash, regex, etc) info from the dir property. I'm talking the path.parse approach?

[minecrawler]

Here is a way to get the filesystems on Linux: https://www.cyberciti.biz/faq/linux-command-list-mounted-devices-in-terminal/

tl;dr: read the file /proc/mounts. On each line, the first bit is the partition (for example /dev/sda3) and the second bit is the mount-point (for example /). You will have to check the given paths against the mount-points and find the closest match (be careful to not only always find the root-fs)

[pstoev]

@minecrawler that's only for linux. path.parse seems more universal to me across different OS's

[minecrawler]

@pstoev yeah, true, but you cannot parse a path on linux in order to get the mount point. Any directory could be located on any disk and partition. There is no way to tell just by looking at the path, as is the case for Windows (I don't know about macOS, however it is *NIX at heart, so it might have the same problem)

[zkochan]

We should also come up with a way to configure the path to the store on file systems. I think we can use a similar format to the one that is used for storing registries and auth info.

store = D:\data\pnpm-store
D:\subdir:store = D:\subdir\foo\pnpm-store
D:\:store = D:\store

[tunnckoCore]

Actually, does fixing that issue means that it will be allowed cross-device linking? If yes, ping me, because don't want to work and store on same device ;d I just have totally separated partition for caches, backups and such things.

[zkochan]

No. Cross device linking using hard links is impossible. The store will be
always on the device on which you run install

On Sat, Apr 29, 2017, 00:32 Charlike Mike Reagent [e-mail]
wrote:

Actually, does fixing that issue means that it will be allowed
cross-device linking? If yes, ping me, because don't want to work and store
on same device ;d I just have totally separated partition for caches,
backups and such things.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pnpm/pnpm/issues/712#issuecomment-298111591, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1pm7jdBl1wEjCtscr1RL4xzHhNt8uNks5r0lrQgaJpZM4NEsZs
.

[tunnckoCore]

@zkochan that's very sad ;/ One more thing why won't use it. Maybe one day when I dont have separate partitions, but.. It's not the only problem. More important _kinda_ problem for me is that non-flat tree, because it messes with my planned package development workflow. Yea, otherwise, it is awesome and super fast exactly because the hardlinks and non-flat architecture.

I just kinda like the trick that i found using flat mode. Let me explain the plan, it's a bit offtopic, but shows the cases/scenario why some like me won't use it at el.

1. I'm using StandardJS, Prettier and Atom with Linter-ESLint and Prettier-Atom packages
2. I have .eslintrc.json file in my packages
3. My packages just includes standard as dependency
4. the eslintrc file (notice that it isn't even eslintConfig field) is intentional, it gives great IDE feeling
   
   
   
   • first, because the Linter ESLint recognizes it
   
   
   • second, allows contributors to just use their global eslint, instead forcing them to have standard globally or forcing them to run npm scripts of the package (no matter that i have script that executes standard)
   
   
   • third, extensibility - for things like enforcing standard/object-curly-even-spacing: error, always, just because hate to not have parens which is allowed in standard by default
   
   
   • fourth, it resolves the eslint-config-standard that is installed by the standard package
   
   
5. My npm scripts are fantastic, they just not allow even to be commited something that is wrong
   
   
   
   • and actually i even don't need them, exactly because that integration that i'm trying to accomplish
   
   
   • everything is realtime in the Atom.. even don't need to run any scripts, because linting and formating is done on save
   
   

So, hope that clear the situation a bit.

[zkochan]

I don't understand what is the issue. It will still use not more space per partition than npm or yarn.

Also I don't see in your workflow where you leverage the flattened node_modules structure. It seems you just use global packages

[tunnckoCore]

Also I don't see in your workflow where you leverage the flattened node_modules structure.

Atom Editor's ESLint resolves eslintrc which in turn will try to resolve eslint-config-standard and if I use Pnpm the Linter-ESLint will throw an error that it can't resolve the StandardJS config because the tree isn't flat. To fix that, one solution is to add eslint-config-standard to my devDeps too, but I don't. If i add it, I'll just add all the eslint-* stuff instead of standard. But I don't want to add them, because the GreenKeeper bumping spam, so i always try to have as low deps as possible.

It seems you just use global packages

No. Almost never, just have rollup, mocha (hence, didn't write mocha for years ;d) and standard just for my fast fingers.

It will still use not more space per partition than npm or yarn.

yea, in any way, in my case it won't be so much, and so i can reconsider using Pnpm one day when i'm tired of yarn.

In any way, keep doing the good job! :tada:

[zkochan]

Should we rename "global store" to something else to reflect that it is per filesystem/partition?

Maybe "partition store", or "disk store"... any ideas?

[AvailCat]

For Linux, maybe this? https://www.cyberciti.biz/faq/linux-unix-command-findout-on-which-partition-file-directory-exits/

I don't know how yarn works

[zkochan]

Well yarn does not use hard links pointing outside of the node_modules folder, so they don't have this issue

[minecrawler]

@Meeeeow Don't use commands/3rd party programs from within your source code. You never know if they exist (even though the GNU utils are popular, there are other options available!). Read the source of truth yourself (/proc/mounts). I already posted a guide earlier.

[zkochan]

I suggest a quick fix before the smart, harder one.

If there is a "cross-device link not permitted" error when creating the hard link from the global store to the project's node_modules then pnpm prints a warning and falls back to copying.

[zkochan]

A temporary fix published with v0.69.0-beta.3. It can be installed via npm i -g pnpm@next

The fix is: if linking from store fails, the package is copied.

Tested on Linux, let me know if there are issues on other systems

[TigersWay]

As this does not seem to be fixable quickly, the "fix" should be added and explained in the first page. Quite a lot of dev. already gave up and it's a pity!

[zkochan]

Mentioned these issues in the README

[zkochan]

Published in v1.15.0. Each disk has a dedicated store now.

[dandv]

Admittedly I haven't fully read each comment on this issue, but "Each disk has a dedicated store now." doesn't seem to be the case with TC drives mounted on Linux.

I've just installed pnpm and haven't configured a store. The README states

If there is no homedir on the disk, then the store is created in the root. For example, if installation is happening on disk D then the store will be created in D:\.pnpm-store.

I have some packages on a TC drive mounted at /t/, and if I run pnpm install there, I get the EXDEV: cross-device link not permitted warning, and a node_modules/.registry.npmjs.org directory is created in each package directory, containing directories and files (not links) for each package. I don't have a home directory on that disk, but /t/.pnpm-store was not created in the root of the TC disk. Instead, ~/.pnpm-store was created.

Presumably can't tell that the /t/ path is not the same disk as /home/dandv?

I tried as a workaround to set npm config set store /t/.pnpm-store and delete /home/dandv/.pnpm-store, but that didn't help - node_modules/.registry.npmjs.org was still created in each package directory, and it contained directories and files (no symlinks), not saving any space.

[zkochan]

@dandv the second time, if you specified store to be on the same disk, you did not get the EXDEV: cross-device link not permitted warning, right? In that case files in node_modules/.registry.npmjs.org are hard links and symlinks. They are not copies of files from the store

[dandv]

Thanks @zkochan. indeed, I didn't get any warnings the second time.

As an experiment, I installed the dependencies from the same package.json twice, in two different directories on the TC drive. Still, 9MB of space were consumed. This is much better than yarn, but I'm curious why 9 extra MB.

/t/prg $ pnpm config get store
/t/.pnpm-store

/t/prg $ rm -rf $(pnpm config get store)

/t/prg $ df -h
Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  368M  986M  28% /t
[...]

/t/prg $ (cd monitors ; pnpm install)
Packages: +372
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Resolving: total 372, reused 0, downloaded 372, done
Running install for registry.npmjs.org/utf-8-validate/4.0.0, done
/t/prg/monitors/node_modules/influx-utils -> ../influx-utils
[...]
/t/prg/monitors/node_modules/utils -> ../utils
dependencies:
+ big.js 5.0.3
[...]
+ simple-xmpp 1.3.0

devDependencies:
+ eslint 4.17.0
+ eslint-config-airbnb-base 12.1.0
+ eslint-plugin-import 2.8.0

/t/prg $ df -h
Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  475M  880M  36% /z
[...]

# At this point, /t/.pnpm-store is ~65MB and monitors/node_modules is ~47MB.

/t/prg $ mkdir monitors2
/t/prg $ cp monitors/package.json monitors2/
/t/prg $ cd monitors2; pnpm install ; df -h
Packages: +372
Running install for registry.npmjs.org/utf-8-validate/4.0.0, done
/t/prg/monitors2/node_modules/influx-utils -> ../influx-utils
[...]
/t/prg/monitors2/node_modules/utils -> ../utils
dependencies:
+ big.js 5.0.3
[...]
+ simple-xmpp 1.3.0

devDependencies:
+ eslint 4.17.0
+ eslint-config-airbnb-base 12.1.0
+ eslint-plugin-import 2.8.0

Filesystem              Size  Used Avail Use% Mounted on
[...]
/dev/mapper/truecrypt1  1.5G  484M  871M  36% /z
[...]

# Only 9MB extra were consumed.