Yarn: Yarn workspaces: a lot of `node_modules` are missing in workspace

[ghost]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I have 2 workspaces. In the root dir I created package.json:

{
    "private": true,
    "name": "project-name",
    "workspaces": ["api", "frontend"]
}

When I use yarn install and I go to the frontend/node_modules I see just several modules, but not all needed which are in the root dir. As far as I understand there's no symlink created for these packages? For e.g. I need ember-pikaday, but it's not in the frontend/node_modules, but it's in the root node_modules.

In short: in frontend/package.json I have ember-pikaday, but it's not in the frontend/node_modules after yarn install. Because of this I can't start my project (Node.js can't find ember-pikaday and thinks it's not installed).

What is the expected behavior?
To get all needed node_modules inside workspaces.

Please mention your node.js, yarn and operating system version.
Node: 7.10.1
Yarn: 1.0.2
macOS

P.S. This feature is AMAZING!

[BYK]

Hey! This is by design, Yarn hoists all dependencies it can to the workspaces root. Why do you need them inside?

[ghost]

@BYK Hi, but how to fix this issue? I can't start project, because it can't find for e.g. ember-pikaday it thinks it's not installed...

[BYK]

@bsvipas this is not really "fixable" I think. The tool itself would need to check if it is available on the higher level. Can you run things from the workspace root?

[ghost]

@BYK I can't do that, because I'm using ember build and in such case there's no point to use workspaces if they are not "smart" enough. I think better is maybe to create symlinks in workspace node_modules which will fix such case issues, because they will know where is for e.g. ember-pikaday package, etc.

EDIT:
But weird thing is my API workspace is smart enough to search node_modules in root, but frontend can't find it... Maybe it's, because of Ember fault?

[ghost]

Basically this happens to me https://github.com/yarnpkg/yarn/issues/4081 and I think this is only with Ember...

[ghost]

@BYK This issue is related with ember-cli-dependency-checker as you said. ~I fixed it and right now I need to polish it and I will try to make PR.~ Seems this is harder than I thought.

[rulonder]

Similar issue in create-react-app, what is the problem with the creation of symbolic links in the local packages?
see https://github.com/facebookincubator/create-react-app/issues/3031

[arcanis]

Symlinks have problems.

[ghost]

I think this is not Yarn fault, it's because of tools which search for node_modules by themself to show you a error, some info or other stuff and it doesn't go in upper level, that's why. Because Yarn with workspaces install almost all node_modules in project dir, not the workspace dir. I tried to use it with simple project (with express) and it works :)

[rulonder]

@arcanis thanks for the link. @bsvipas yep, it seems that is the create-react-app is guessing the package absolute path.

[lostpebble]

This is an issue I'm running into with various tooling. It's not really yarn workspace's fault - it makes sense to push the modules to the parent directory. But it can be frustrating when working with and mixing your other workspace modules as regular node modules within each other.

The biggest issue I have is that using auto-import in the IDE's I've been trying (WebStorm and VS Code), trying to auto-complete a class / function from a different workspace package - as in the OP example, something from api inside of frontend - the tooling will decide to create a relative path to the other workspace module, like: import { getUser } from "../../api" - instead of using the node module resolution of just import { getUser } from "api".

If there was instead a symlink to the module in the actual workspace folder (only needed for workspace modules) perhaps this could be avoided. As you can open up a new project window in only the folder of each workspace module, which should scope it only to that directory, and check in that node_modules folder instead of going further up and creating a relative path.

[ghost]

@lostpebble I can just agree with you. If symlinks was created in all workspaces from the root (project node_modules) all issues will be gone and we can all easily use Yarn workspaces with any tool. Maybe we need to open new issue to suggest by adding symlinks?

[gustaff-weldon]

@bsvipas we also use monorepo with interconnected Ember addons and we solved the issue by automatically symlinking whatever package specifically requires from a hoisted node_modules folder into the package/node_modules folder.
Happens after bootstrap. Works so far.

[MarkKahn]

FYI I'm running into this same issue with babel plugins. I'm getting:

ReferenceError: Unknown plugin "...../node_modules/babel-plugin-transform-flow-comments" specified in "base" at 1, attempted to resolve relative to "<child path>"

Since babel won't look outside of the project folder but the yarn workspace is installing everything in <parent path>/node_modules. Symlinking in the child module resolves it which is what I'm doing for now. I believe webpack will have the same issue but I've just started looking at workspaces.

To the various comments in this thread arguing that it's not yarn's fault, I would argue that it is. Yarn workspaces change the expected directory structure that has been consistent since whatever npm version was with node v0.x (for at least top-level modules)

[joefiorini]

I'm having this issue as well, in my case with babel. We have a monorepo with a couple rails apps, each of which use webpack to bundle react code. What I don't understand is that even prior to workspaces we had babel specified at the root of the project, and it would get loaded fine because the standard node resolution rules would look up from the rails app's folder to the root and find babel.

I'm quite confused as to how this could change when using workspaces, but it _seems_ like every case mentioned here should work just fine thanks to the default resolution rules. No?

[f-roland]

I've had this issue too, but not consistently, so I couldn't really pinpoint the issue... My best guess is that if one of the packages shares all its dependencies with the other packages, they're all hoisted in the root, and that package doesn't even have a node_modules folder.

it's mostly a problem for binaries that are run form the package with a script pointing to the package's node_modules folder, so I added this script as a prepare hook in the main package.json file. Basically, if one of the package is missing a node_modules folder, I create it and then symlink the node_modules/.bin folder from the workspace root.

// scripts/link-binaries.js
const { readdir, access } = require('fs');
const { promisify } = require('util');
const { resolve } = require('path');
const shell = require('shelljs');
const { map } = require('ramda');

const asyncReaddir = promisify(readdir);
const asyncAccess = promisify(access);

const packagesDir = resolve(__dirname, '../packages');
const rootDir = resolve(__dirname, '../');

function linkBinaries(_package, nodeModulePath) {
  console.log('linking binaries in ', _package);
  shell.mkdir(nodeModulePath);
  shell.exec(`ln -s ${rootDir}/node_modules/.bin ${nodeModulePath}/.bin`);
}

async function perform() {
  const packages = await asyncReaddir(packagesDir);

  const result = map(async _package => {
    const nodeModulePath = resolve(
      packagesDir,
      `./${_package}`,
      './node_modules'
    );
    try {
      return await asyncAccess(nodeModulePath);
    } catch (e) {
      if (e.code === 'ENOENT') {
        linkBinaries(_package, nodeModulePath);
        return;
      }
      throw e;
    }
  }, packages);
  return Promise.all(result);
}

const successHandler = () => {
  console.log('Binaries linked !');
  process.exit(0);
};

const errorHandler = err => {
  console.error(err);
  process.exit(1);
};

perform().then(successHandler, errorHandler);

// in root package.json
"scripts": {
  "prepare": "node scripts/link-binaries.js"
}

the script could probably be improved, but it fixed the issue for me

[Zerim]

Also having issues with this where hoisting is leading to a number of difficult to resolve bugs in our Typescript setup. In some cases, it's causing duplicated conflicting module declarations and in other cases it's causing it to not find types that would be declared by a peer dependency. The inconsistency of the behavior makes it difficult to address by adjusting the global compiler settings in tsconfig.json.

[Zerim]

In case it helps anyone else, a hack that seems to work to keep your modules from being hoisted is to install a conflicting version of the same module in the root directory of your monorepo. That way the specific version used by your package will be kept in your respective workspace.

For my issue mentioned about the fix was a bit different, I used the resolutions field to force a single version of my dependency to be installed and in that way I avoided conflicts between the hoisted version of the dependency and the local versions of the dependency in the workspace.

[blackxored]

Tools like react-native link and react-native-git-upgrade seem to have this problem as well.

[darthtrevino]

react-vr is having problems with this setup as well. It would be nice to be able to opt-out of hoisting on a per-package level or mark dependencies as being required-local.

[mjsisley]

Perhaps adding a shell node_modules directory to each workspace that mirrors the directory structure of the root node_modules directory and has 1 file that re-exports the entry in the parent would solve this issue?

This gives the advantage of the root node_modules, but allows any library that is expecting to find a node_modules folder in its directory to keep working.

While ideally babel and other tools would each be useable with yarn workspaces without this need, it seems like it may be easier for yarn workspaces to provide a way to conform to the many tools that have the expectation of having a node_modules folder.

This could be opt in via a nodeModulesInWorkspace: boolean|array. (true to turn it on for all workspaces, or a whitelist of workspaces that need it).

[tilwinjoy]

Maybe the external tools (ember-cli, webpack etc) should travers upwards until it finds a node_modules before throwing errors?

[PaulRBerg]

I used to have a similar problem when ussing yarn 1.9.4 and trying to install eslint-config-airbnb-base. I was getting the following error when running eslint:

Cannot find module doctrine

I initially solved it by adding doctrine to {"resolutions"} in package.json, but I started to get other quirky warnings and I generally felt tihis was too hackish for what I wanted to achieve (get a linter to work).

But then I upgrade from yarn 1.9.4 to 1.15.2 and that solved the problem. I'm on macOS Mojave 10.14.2.

[traviswimer]

This is an old issue, but I have found a solution if anyone runs into this. Yarn has a somewhat poorly documented nohoist feature:
https://yarnpkg.com/blog/2018/02/15/nohoist/

It lets you specify packages that you don't want to have hoisted to the top. For example, if you want to completely eliminate the node_modules hoisting, you could use something like this in your package.json:

{
    "workspaces": {
        "packages": [
            "projects/*",
        ],
        "nohoist": [
            "**"
        ]
    },
}

To get this to work, I had to delete my problematic node_modules directory and then run yarn install.

[leolcao]

I also meet this problem.
If not set nohoist for yarn workspaces, all packages are stored in the root node_modules folder, when I run yarn workspace app-a build, it even cannot resolve native node.js modules, like:

Cannot find name 'console'

But, If I enter the workspace package folder(app-a), and run npm run build it works.
After set the nohoist from @traviswimer comment, I can run yarn workspace app-a build now, but the drawback is also very big: there are duplicated node_modules in each workspace package folder.

At the end, I give up, and will try yarn v2 workspace.