Yarn: nohoist to ignore a package instead of ignore modules

[huchenme]

Hi,

I have a repo with a few sub packages, one of them (package1) got react-scripts 1.1.0 which use older version of Jest, babel and babel-jest, another sub package (package2) added dependency of newer version of Jest, babel and babel-jest.

If I use

"nohoist": [
   "**/react-scripts/",
   "**/react-scripts/**",
]

Jest, babel and babel-jest won't be hoisted at all to the root which make the packages using package 2 breaks.

If I don't put nohoist option, for some reason the hoist packages are the lower versions from react-scripts.

I am not sure what is the best option to resolve this, but ideally I just want to have package 1 nohoist (keep it's sub packages within its own node_modules)

[connectdotz]

try:

"nohoist": [
   "package1/react-scripts/",
   "package1/react-scripts/**",
]

[huchenme]

@connectdotz it does not work with that. As react-scripts got babel-core, babel-jest and jest dependencies. Those dependencies will not be hoisted to root if I apply above code.

What I want to achieve is that the dependencies in package 1 stay in its own node_modules, the other packages hoist as normal

[connectdotz]

if you want to disable hoist for all modules under package1, you can simply use

nohoist": [
   "package1/**"
]

if you want to disable specific modules under package1, such as react-native, use the one I mentioned above.

nohoist just perform pattern match, there is no black magic, feel free to experiment...

[huchenme]

@connectdotz Thanks for the reply, but it is still not working.

By using

nohoist": [
   "package1/**"
]

my package2 will not hoist all dependencies that are under package1/**.

Previously if it was installed without the nohoist option, they were all hoisted to root node_modules, but with nohost set to package1/**, anything included in react-scripts in package1 or dependencies of dependencies of react-scripts
will not hoist to root in package2

My expectation

nohoist": [
   "package1" (or anything we call it)
]

should not affect package2, it should just not hoist dependencies in package1 but keep existing behavior of package2 and other packages unchanged.

[connectdotz]

should not affect package2, it should just not hoist dependencies in package1 but keep existing behavior of package2 and other packages unchanged.

That's exactly what "nohoist": ["package1/**"] does... make sure your nohoist config is in the root package.json and not in the workspace's. What you asked for is not unusual and it should work if configured correctly. If you still have any problem getting it to work, it is best to create a sample repo so we can help you further.

[huchenme]

@connectdotz

Sure, I have created a repo: https://github.com/huchenme/workspace-nohoist

There are 2 branches: master and nohoist:

In master branch, I have added some dependencies in package2. If you perform a yarn install at root, you could see most packages have been hoisted to root node_modules except 2:

and jest-environment-jsdom is listed in root node_modules as well

In nohoist branch, I have added react-scripts in package1, and add nohoist option at root package.json.

  "workspaces": {
    "packages": ["packages/*"],
    "nohoist": [
      "package1/**"
    ]
  }

After run yarn install, you could see that the node_modules in package2 is changed.

And jest-environment-jsdom is no longer contained in root node_modules

---

jest-environment-jsdom is just one of the things breaks for me, what I really expected is that in nohoist branch, the node_modules at root and node_modules in package2 should be the same as master branch if I put nohoist to package1/** or anything we want to call it.

[connectdotz]

great, now we can have more concrete discussion...

You mentioned 2 issues:

1. package2 node_modules is different between master and nohoist branch.
2. some modules, such as jest-environment-jsdom, has not been hoisted to root.

I can now see your logic and thinking... you noticed, after introducing a new dependency "react-scripts": "^1.1.4" in package1, yarn changed its hoist decision (due to version conflict between package1 and package2), and thought: if I tell yarn nohoist package1, package2 should then be hoisted as if there is no package1...

It is an intuitive thinking, unfortunately, the current conflict version resolution logic in yarn doesn't take nohoist into consideration (note-1), i.e. when conflict versions are detected, it will always go to the most frequently referenced or the first encountered. So you can see that it could yield different decision when your underlying dependency graph changes or even with new yarn resolution algorithm. Therefore, you should never assume which version will be selected for hoisting during version conflict. But rest assured, yarn, or any package manager for that matter, will always guarantee that modules, who have declared dependencies, will have access to their dependencies either through their local node_modules (not hoisted) or their parents' (hoisted). (node-2).

This leads to your 2nd issue... If your root needs certain version of jest-environment-jsdom, I hope it should be pretty obvious now that you will need to add it to the root package.json. If you were just trying to work around some 3rd party package's issue, you should also make sure they are addressing the underlying problems so you don't have to hack your own package.json...

---

• note-1: conflict version resolution didn't take nohoist into consideration: you can easily see this by disabling the nohoist rule in the nohoist branch, you can see package2 node_modules didn't change at all. Should the conflict resolution take nohoist into consideration? probably... it would be less confusing and more efficient for sure, however, it is not a solution for your issue, i.e. broken dependency graph, as we discussed above.
• note-2: You can see that for jest-environment-jsdom, even though you can't find it directly under root's node_modules, the modules that declared the explicit dependency of jest-environment-jsdom all have it locally installed. You can further see how yarn make these decision via the why commands:

$ yarn why jest-environment-jsdom
yarn why v1.6.0
[1/4] 🤔  Why do we have the module "jest-environment-jsdom"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "jest-config#[email protected]"
info This module exists because "_project_#jest-config" depends on it.
info Disk size without dependencies: "20KB"
info Disk size with unique dependencies: "964KB"
info Disk size with transitive dependencies: "5.21MB"
info Number of shared dependencies: 96
=> Found "jest-cli#[email protected]"
info This module exists because "_project_#package2#jest#jest-cli" depends on it.
info Disk size without dependencies: "20KB"
info Disk size with unique dependencies: "964KB"
info Disk size with transitive dependencies: "5.21MB"
info Number of shared dependencies: 96
=> Found "package1#[email protected]"
info Reasons this module exists
   - "_project_#package1#react-scripts#jest#jest-cli" depends on it
   - Hoisted from "_project_#package1#react-scripts#jest#jest-cli#jest-environment-jsdom"
   - Hoisted from "_project_#package1#react-scripts#jest#jest-cli#jest-config#jest-environment-jsdom"
   - in the nohoist list ["/_project_/package1/**"]
info Disk size without dependencies: "16KB"
info Disk size with unique dependencies: "960KB"
info Disk size with transitive dependencies: "5.21MB"
info Number of shared dependencies: 96
✨  Done in 3.37s.

[huchenme]

What you are saying is true, but I am suggesting a feature that if we can disable hoisting for some packages without affecting other packages at all. Introducing react-scripts into package1 have clearly breaks package2 (I could manually spend a whole day to find all packages which break and manually add them but I don't think it should be that way).

Should we consider to change current nohoist behaviour to disable hoisting for certain packages, or add such feature to nohoist? Or can we just disable hoist for all packages but still use yarn as the package manager?

[connectdotz]

Introducing react-scripts into package1 have clearly breaks package2

how? different node_modules structure != break package2. If package2 is wrongly configured, such as has an unclaimed dependency that it plans to consume, why don't you nohoist package2 instead?

Or can we just disable hoist for all packages but still use yarn as the package manager?

yes, If you still want to share the top-level packages, use "nohoist": ["**"]; if you don't care about sharing at all, just don't use workspaces.

Should we consider to change current nohoist behaviour to disable hoisting for certain packages, or add such feature to nohoist?

If you meant conflict version resolution I mentioned above, even if we do fix that, your environment might still break, say when package3 or any other 3rd party library introduced a conflict version... It is not a solution to guarantee package2 will always have the unclaimed dependency intact. I hope you can see the problem is not which version would be hoist, but why will it matter to package2, because it shouldn't...

BTW, the 2 issues you showed in the sample repo are more red harrings, I am still not sure what is the actual problem you referred to in your original post:

Jest, babel and babel-jest won't be hoisted at all to the root which make the packages using package 2 breaks.

which package breaks? package2? how? not finding the package or finding the wrong package? Without knowing what exactly is not working, we might be wasting time barking on the wrong tree...

[mtford90]

I'm having a similar problem.

I have a monorepo that includes both a react web package, and a react-native expo package. The expo package relies on an older version of react to the web package.

To keep things simple, I wanted to prevent ANY dependencies from within the react-native dependent package from being hoisted, and so I tried the following:

   "packages": [
      "packages/*"
    ],
    "nohoist": [
      "**/my-react-native-expo-app",
      "**/my-react-native-expo-app/**"
    ]

However for some reason I still end up with e.g. @expo and react-native and various other dependencies in the root node_modules. This causes various typescript conflicts.

I then tried the following as per your expo example @connectdotz :

      "**/react-native",
      "**/react-native/**",
      "**/react-native-scripts",
      "**/react-native-scripts/**",
      "**/expo",
      "**/expo/**",
      "**/jest-expo",
      "**/jest-expo/**"

However I still end up with conflicts because of differing react versions.

If I add react to the above, I end up with new errors from other hoisted dependencies that depend on react in other packages. And the cycle continues as I add more and more nohoist packages and new errors appear.

Is it not possible to have one workspace maintain its original directory structure as if it's not a workspace i.e. take NO part in the hoisting mechanism - totally ignored - but still have other workspaces as dependencies?

[tommedema]

@mtford90's question is already 5 months old without any feedback. Are people not experiencing issues from this? In my case it seems impossible to use yarn workspaces together with certain modules, especially react-scripts. Having trouble reproducing it in a small repo due to the complexities of the dependency trees with nohoist but it seems related to the above discussion.

I've documented my issue in #7246 but am not entirely sure if they're related

[mtford90]

@tommedema :(

If anybody is interested, the "solution" I eventually ended up with was:

1) Ditch yarn + switch back to NPM.
2) Use lerna to manage the packages, hoisting etc.
3) Use install-local for those packages that can't handle sym links e.g. react native with the metro bundler.

I look forward to the day that yarn workspaces have the flexibility needed to handle these situations.

[arcanis]

I said it before, I'll say it again: nohoist is a workaround. If a package needs nohoist to work, then this package is broken and this should be reported to its maintainers. It will eventually break - nohoist won't be ported over to the v2, for example, so it's much better to fix it now than to wait next year.

In this particular case someone needs investigate how to fix the problems in RN that make it rely on hoisting that much. This is the real fix - nohoist isn't.

[huchenme]

@arcanis, I might have missed your message but I have not seen that message in this thread before. I spent so much time in this thread to create repo and explain why we need a feature, I my opinion, nohoist feature is incomplete at the moment that is why we are requesting a feature a make it easier to use.

If nohoist option is not right at the first place, I think we should remove this option in yarn, as you said, the package breaks and people should fix the problems themselves

Additionally, I don’t think anything in my example is break and should fix, if you know how to fix it, please let me know

[arcanis]

There are various threads about nohoist. It's a confusing feature even for myself and I'm not even too sure to understand how it works - the hoisting part of the v1 codebase is quite messy and there's a lot of corner cases there. Part of why we've redesigned all this in the v2 😉

If nohoist option is not right at the first place, I think we should remove this option in yarn, as you said, the package breaks and people should fix the problems themselves

It won't be part of the next major release (even more so since we won't generate node_modules at all). I don't think we'll remove it from the v1 though since some setups depend on it and removing it would be semver-major.

Additionally, I don’t think anything in my example is break and should fix, if you know how to fix it, please let me know

What would be broken according to your post is that Jest cannot detect jest-environment-jsdom if it's being hoisted (is that still the case? I think it should work, btw 🤔). Whether a package is hoisted or not is an implementation detail and you shouldn't assume anything about it. In fact, you shouldn't even assume that there will be a node_modules. The only thing that will always be correct is acquiring dependencies through require and require.resolve.

[mtford90]

Agreed on it being removed if no intention to support these kind of issues. Anyone who tries to use this feature is probably found to hit this roadblock eventually.

A system that relies on the good behaviour (in this case, the good behaviour of library maintainers) is probably destined for failure.

[connectdotz]

A system that relies on the good behaviour (in this case, the good behaviour of library maintainers) is probably destined for failure.

ouch, that hurts, although I am guilty as charged... sorry for not being able to come back to this issue earlier.

~10 months ago, Instead of adding more code to nohoist, I started to wonder if this is a self-inflicted wound due to hoisting, and if there is a better way to "hoist" so we won't break packages thus no need for "nohoist"... Discussed with @arcanis and he mentioned the v2 is on the way that should make this a moot point, I sort of write off this issue from my conscience since...

ok, so a quick fix that makes "version-conflict-resolution works better with nohoist" is possible but I am hesitated to embark on such journey if v2 is on the way here soon or if the core team think we should not support "nohoist" in v1 code base anymore... thoughts?

[mtford90]

@connectdotz that wasn't aimed at you or yarn - but rather at the package maintainers who's packages are causing nohoist to break ;) - e.g. react & react-native. I meant that yarn/nohoist shouldn't depend on the good behaviour of packages that are installed using it.

Sorry for the perceived criticism and thanks for the update!

And yes - sounds like we should all wait until v2 drops.

[arcanis]

A system that relies on the good behaviour (in this case, the good behaviour of library maintainers) is probably destined for failure.

For the record I don't necessarily think of it as a matter of "good" versus "bad" behavior (as tempting as it might sometimes be). It's just that in the past years and until a few months ago package managers weren't enforcing good practices, and if you don't teach then you can't expect your users to learn.

Starting with PnP we've been much stricter than we previously were, and since PnP is the default install strategy from the v2 onwards it will become easier for library authors to spot problematic patterns. It's unfortunately a bit painful in some cases, but I hope the long-term benefit will outweight the cost.

Also note that those projects are just like Yarn in that they usually rely on their community's help to fix issues - so if you're familiar with for example react-native, I'm sure they would be glad if you could investigate what would be required for everything to work out of the box!