Yarn: Yarn unable to execute binaries installed in higher level node_modules/.bin

[lucbelliveau]

Given the following folder hierarchy:

• packageA
  
  
  
  • node_modules
  
  
  • .bin
    
    
    
    
    
    • myprogram
    
    
    
  
  
  • packageB
  
  

yarn run somecommand

from inside packageA/node_modules/packageB fails. The same command using npm works.

The problem showed up because packageB has a postinstall directive that executes myprogram.

I've determined the problem is because yarn doesn't traverse the directory tree looking for additional node_module directories to add to the path.

I'll submit a PR shortly.

[arcanis]

Imo you shouldn't rely on this behavior, whether it's on Yarn or NPM. You should declare packageA as a peer dependency of packageB instead, which would make Yarn add a symlink to packageB/.bin/myprogram that would target packageA/myprogram, effectively making myprogram accessible from packageB scripts.

To prove to you that the original behavior you describe is unreliable, just consider that the hoister might choose to hoist packageB at the same level as packageA, in which case there will be no link between their node_modules at all.

Does something prevent you from declaring packageB as peer dependency of packageA?

[lucbelliveau]

I was under the impression that yarn should mimic, whenever possible npm's behavior to ensure it can be used as a drop-in replacement?

For the sake of argument I tried moving the offending package into the peerDependencies of packageB to test, and it still fails with a command not found error...

EDIT: I just tried putting packageB as a peerDep of packageA with the same result.

[hoodie]

I have a similar issue on windows where I run yarn clean (rimraf) in a workspace as postinstall. Yarn executes scripts from the wrong location and therefore does not find them.

example

.
├── modules
│  └── my_package
│     └── package.json
└── node_modules
   └── @company
      └── my_package
         └── package.json

my postinstall script is executed from within node_modules/@company/my_package and not modules/my_package, so the .cmd looks for a location outside of the root folder.

@lucbelliveau is that what you meant? otherwise I'll open a new issue

[lucbelliveau]

@hoodie yes that is what I mean... PR #4642 should fix your issue as well...

[petemahoney]

Just thought I'd share in hopes this will support the need to merge the fix that @lucbelliveau has made. Also to see if others think what I describe is in fact the same problem described in this ticket.

Using yarn v1.1.0, just ran into what appears to be the same problem described here. Ran yarn install --verbose on myProject which had a dependency on packageB which has a postInstall command (typings install) where typings is a devDependency of both myProject and packageB and as such has an executable command path set up in myProjectnode_modules\.bin - the result is the error below:

verbose 52.355 [pathToMyProject]node_modulespackageB: Command failed.
Exit code: 1
Command: typings install
Arguments:
Directory: [pathToMyProject]node_modulespackageB
Output:
'typings' is not recognized as an internal or external command,
operable program or batch file.
at MessageError (C:\Program Files (x86)\Yarn\lib\cli.js:139:5)
at ProcessTermError (C:\Program Files (x86)\Yarn\lib\cli.js:158:1)
at ChildProcess.proc.on (C:\Program Files (x86)\Yarn\lib\cli.js:29955:15)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:877:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
error [pathToMyProject]node_modulespackageB: Command failed.
Exit code: 1
Command: typings install
Arguments:
Directory: [pathToMyProject]node_modulespackageB

For the moment our team is just going to install typings globally (and yes for those that know typings is depreciated, we'll eventually get rid of that altogether but don't want to muddy the waters here with that)

Appreciate your feedback @lucbelliveau :)

[lucbelliveau]

@petemahoney It definitely sounds like the same issue...

[hoodie]

@BYK, could you perhaps review @lucbelliveau's PR #4642 ?
thank you

[BYK]

I was under the impression that yarn should mimic, whenever possible npm's behavior to ensure it can be used as a drop-in replacement?

This is not correct. We aim to match behavior that is not a bug or an edge case and this looks like an edge case or even a bug to me.

@petemahoney - the issue you are facing is expected since the packages are not on the same tree. If a package needs to use another package, it should list the package in its dependencies in some form (dev, peer or normal dependency). It doesn't sound like that's the case for you.

[BYK]

@lucbelliveau

EDIT: I just tried putting packageB as a peerDep of packageA with the same result.

Looks like it should be the other way around. Since packageB needs myprogram which is provided by packageA, packageB should list packageA as a peer dependency.

[BYK]

Since this is not a bug, closing the issue. You can file a new issue if there are problems with the script running when listing dependencies properly via peer dependencies or by other means.

[lucbelliveau]

@BYK I don't agree... I'm concerned my use case isn't being understood correctly. Let me try again, last time. :)

webpack-custom-plugin has a devDependency on gulp
webpack-custom-plugin has a postinstall directive that executes the gulp binary.
my-great-react-app has a devDependency on webpack-custom-plugin.
yarn install fails.

A valid peerDependency in this situation would be webpack; because my-great-react-app uses webpack and presumably my webpack-custom-plugin works with specific versions of webpack. The authors of webpack-custom-plugin for some reason want to use gulp in the postinstall of their plugin. (In my case it's actually the virtualenv package, and postinstall needs to run virtualenv-postinstall)... but we end up with the same problem.

There is no reason for my-great-react-app to install gulp IMO...

This situation works perfectly fine using npm ... but yarn will not work, unless I either install gulp globally, or pollute my dependencies with something I am not using, and won't be used except during installation.

If you still believe this is an edge case, and not how yarn should work, I'll have to come up with a workaround... but IMO this is not what peerDependencies are for.

[arcanis]

I think the best way to help us understand what's the problem is would be to set up a reproducible repository somewhere with the dependencies that cause the issues, and only them. Right now reading this thread gives me the impression that we are confusing three or four different topics :)

[amitic]

I agree with @lucbelliveau on this.
We've moved to yarn (not that) recently, but had to move back to npm because of this issue.
I really hope this gets reconsidered, reopened and ultimately resolved.

@arcanis Here's an example where yarn fails: https://github.com/amitic/yarn4640-a

• "packageA" from @lucbelliveau's example is https://github.com/amitic/yarn4640-a
• "packageB" is https://github.com/amitic/yarn4640-b

Both have the same devDependencies and scripts (although just one of cpx/rimraf would have sufficed):

  "scripts": {
    "clean": "rimraf dist && mkdir dist",
    "postinstall": "yarn run clean && yarn run build",
    "build": "cpx index.js dist"
  },
  "devDependencies": {
    "cpx": "^1.5.0",
    "rimraf": "^2.6.2"
  }

packageA also has packageB as a dependency:

  "dependencies": {
    "yarn4640-b": "git://github.com/amitic/yarn4640-b.git"
  },

packageB's scripts work ok on it's own, but when postinstall is run while adding it as a dependency to packageA, yarn fails with (running on windows):

'rimraf' is not recognized as an internal or external command

In practice, we've had yarn fail on scripts using webpackand @angular/compiler-cli when those packages are used in a dependency's postinstall script.

[hoodie]

@amitic I see the same thing and it's a blocker for our switch to yarn workspaces.
Perhaps this deserves an issue of it's own, because the original text by @lucbelliveau was not clear enough to convince @BYK
This is still an open issue in yarn 1.3.2

We also see this in production.

> npm run clean
...
> rimraf lib

module.js:471
    throw err;
    ^

Error: Cannot find module 'C:\build\node_modules\rimraf\bin.js'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.runMain (module.js:604:10)
    at run (bootstrap_node.js:389:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:504:3

[amitic]

I just gave @lucbelliveau's PR (#4642) a try and it seems to work ok in at least 2 cases (the one in the demo repo and another one we had that was failing with a dependency invoking @angular/compiler-cli's ngc inside the postinstall script).

@hoodie While I do hope this gets fixed soon, I'm not really sure the best way forward is to open a new issue that would be an obvious duplicate. Let's call that a plan B in case the devs miss this. :)

[BYK]

Reopened for further investigation and discussion. I still think #4642 is too broad and is prone to errors but we certainly don't want to have issues with scenarios like https://github.com/yarnpkg/yarn/issues/4640#issuecomment-339648272

[lucbelliveau]

Since it works using npm, I implemented the same behaviour in #4642 that npm uses... If anyone has a better suggestion I am willing to implement...

[Antontelesh]

Faced similar problem today when using yarn workspaces.

My setup:

/packages
  /package-a
  /package-b

/packages/package-a/package.json

{
  "name": "package-a",
  "version": "1.0.0",
  "dependencies": {
    "package-b": "^1.0.0"
  }
}

/packages/package-b/package.json

{
  "name": "package-b",
  "version": "1.0.0",
  "scripts": {
    "clean": "rimraf lib",
    "build": "tsc --build tsconfig.json",
    "postinstall": "yarn clean && yarn build"
  },
  "devDependencies": {
    "rimraf": "^2.6.2",
    "typescript": "^3.1.6"
  }
}

When running yarn install from the root, it fails on postinstall hook on Windows, saying something like

Error: Cannot find module 'C:\project\node_modules\node_modules\rimraf\bin.js'

On a mac it works properly, fails only on Windows. And it doubles the node_modules part in the path for some reason. So it expects a folder node_modules to have another node_modules as a direct child.

[alfaproject]

This is especially important in a monorepo where all my dev dependencies are installed at the root so that I don't duplicate them everywhere.

[arcanis]

This error seems entirely different from the one initially reported. Can you open a separate issue and add a repro step?

Also note that I don't develop on Windows, so the best chances to get this bug fixed fast is to either open a PR or at least make sure that your post contains enough information to understand where the problem starts (which might involve adding console.log in your Yarn binary to pinpoint the location).