Yarn: [feature-request] linking binaries from the root in workspaces

[akosyakov]

I wonder whether it would be possible to link binaries in all workspaces from dependencies declared in the root?

Right now to call binaries within a workspace one should redeclare it. It leads to trouble with keeping identical versions between workspaces.

[bryzaguy]

+1 Yes this!

[bryzaguy]

Basically create the symlinks in the .bin folder of each workspace project for the matching hoisted dependencies that project relies on. I have to script this right now to keep everything working.

[rtm]

@bryzaguy But I thought that was already how it worked, assuming you are referring to the links within node_modules/.bin. At least if I understand the question correctly. Although at the moment I am running into some issue with this behavior; see https://stackoverflow.com/questions/46482966/yarn-putting-broken-links-into-node-modules-bin-in-workspace-repos.

Or is this issue requesting that links be placed in all packages even for dependencies only declared at the root? That seems questionable.

[akosyakov]

Or is this issue requesting that links be placed in all packages even for dependencies only declared at the root? That seems questionable.

I concern mostly about managing dev dependencies in one place.

BTW as a workaround one can run yarn in a workspace before running scripts. It links binaries then into the workspace.

[wojtekmaj]

This would greatly improve install time, disk usage and performance in my use case. This is one feature that sadly stops me from using yarn in most of my projects I develop.

[julien-f]

IMHO, if your binaries are used directly from your packages, they should be declared as dependencies of these package, not simply of the root.

I personally try to keep some locality, root dependencies are only used from the root (https://github.com/vatesfr/xen-orchestra/blob/master/package.json).

[AviVahl]

This is a small feature that can go a long way in terms of usability.

I have a repo with several packages, where I'm using yarn workspaces for proper cross-package development.

Many of my CLI-based devDependecies (typescript, rimraf, mocha, karma, tslint) keep repeating in every package, and I would rather control their versions in a single place (the root package.json).

If I don't include them in every package.json (not just the root one), scripts in those packages aren't able to access them. I end up repeating these deps and their versions just so the binaries are accessible.

There are probably several ways to address this...

Linking the root .bin doesn't seem like a good idea, just for those cases where the specific package has a unique dependency (meaning one not moved to the root scope) with a cli.

Maybe yarn run [script] should automatically access the higher-up .bin folder as well, just in case a cli isn't available in the current level's .bin? Maybe it should work this way just for workspaces?

[rally25rs]

Note this is affected by #4730.
As of v1.3.2 bin links seems to only be at the root level (not sure what happens if packages have conflicting versions.)

Also yarn run from any individual package is broken in 1.3.2 because bin links are no longer there.

Maybe yarn run [script] should automatically access the higher-up .bin folder as well, just in case a cli isn't available in the current level's .bin?

this seems like the real answer to me. The PATH set up by yarn run and yarn exec should have the package's .bin, then the root .bin (then the rest of the system paths).

[rtm]

Maybe yarn run [script] should automatically access the higher-up .bin

Not everyone uses yarn run for everything. I and many other people use makefiles, which expect the executables to be in $ROOT/packages/$PKG/node_modules/.bin, even if the package itself has been hoisted and installed in $ROOT/node_modules. If they are not, our entire make process would break instantly.

Overall, this is a bad idea. I don't warn yarn guessing about my intentions. If I want a dependency in a sub-package, then I will put it there. If I want to update dependencies in multiple subrepos, then I will do that. Any functionality along the lines proposed here should be the responsibility of some tool running on top of yarn, for instance a tool which would allow you to update the dependency version in multiple sub-packages. Dependencies at the top level should mean exactly one thing, which is a dependency of the entire monorepo, such as oao.

[rally25rs]

Maybe yarn run [script] should automatically access the higher-up .bin

Perhaps a wording difference;

I submitted a PR above that adds the root's node_modules/.bin to the env PATH _after_ the individual package's package/{pkg}/node_modules/.bin so that a yarn run command should first use any bin linked in itself, but all-back to the root level as the 2nd option in the PATH. The anticipation is that at some point another PR would put the bin links in the individual packages again, not only put them in the root.

Using PATH to check the package then the root is the way I interpreted "automatically access the higher-up .bin" but I can certainly see a difference in interpretation there.

I do think that each package's dependencies should be bin linked in their own packages/{pkg}/node_modules/.bin (which they are not currently or in v1.3.2). If all the bin links are also in the root then I don't think that would cause any issues, but to me it seems like the bin links should only be in the root if the root package.json contained that dependency.

[rtm]

If it is true that 1.3.2 does not bin link dependencies in their own directories, that seems like something between a severe bug, and a major design flaw, and a huge breaking change. What am I missing?

I would try it but am wary of breaking my system by installing 1.3.2. The release page contains no changelog-like info for 1.3.2 so I'm having trouble finding any documentation of this change. I cannot see any commits in the range in question that appear to related to this issue.

[rtm]

This would greatly improve install time, disk usage and performance in my use case.

@wojtekmaj I fail to see how. Let's say I have a TypeScript dependency in subrepo1 and subrepo2., so I can get subrepo1/node_modules/.bin/tsc etc. But that is just a symlink to ../../node_modules/.bin/tsc, and TypeScript itself is still hoisted and takes no additional install time, disk space, or performance. What am I missing about your use case?

I sort of understand the notion that you want the bin linking in subrepos but yet not have to worry about changing versions in multiple places, but what if for some reason you do want different versions? More basically, having yarn put things magically in some subrepo's node_modules/.bin directory just because they happened to be specified at the root violates the principle of least surprise and the definition of how things get into node_modules/.bin, which is that packages in that repo put them there.

[wojtekmaj]

@rtm Maybe I did not fully understand. I have the following situation:

-package
 -node_modules
 -subpackage
  -node_modules

and in subpackage's package.json file I'm including package like so:

"package": "../"

In such case, npm does a symlink. Yarn, on the other hand, installs everything again in package/subpackage/node_modules/package directory, doubling the amount of space needed for the modules.

[rtm]

@wojtekmaj Now I'm really confused. Why would you add the overall, top-level package as a dependency to a sub-package (workspace)? What is your ultimate objective? Perhaps you could show some things that you want to work but aren't now or not in the way you hoped. The following should work just fine:

-package

  -node_modules
    -.bin
      -tsc
      -oao
    -typescript (hoisted)
    -oao (becasue specified as dependency at top-level)

  -package1 (specifies typescript as dependency)
    -node_modules
      -.bin
        -tsc -> ../../../node_modules/.bin/tsc
        -oao: NOT here because is was not specified as a dependency in this workspace
      -typescript: NOT here because it was hoisted
      -oao: NOT here because it was specified at top level, not this workspace

  -package2 (similar)