Berry: [Bug] Yarn PnP behaves different than Yarn in workspaces when package.json scripts calling a dependency

[DimitrK]

Describe the bug
We are trying to make a switch from Yarn to Yarn2 + PnP while using workspaces.

In a monorepo setup we place devDependencies such as webpack in root package.json.

All workspaces are located under /apps directory where they all have a dedicated package.json file with their runtime dependencies and a build script similar to that one:

{
  "scripts": {
    "build": "webpack --config webpack.config.js"
}

Running in Yarn v1 workspaces yarn workspaces run build from root path, executes the build script for each of the workspaces without a problem.
When switching to Yarn PnP running the same command produces the following error

command not found: webpack

To Reproduce
I created a barebone morepo setup trying out various package management setups and technologies. The two that best describe this issue are the following.

• Yarn with Yarn workspaces playing as expected when PnP is not enabled

  1. checkout https://github.com/DimitrK/monorepo-package-manager
  2. run yarn install
  3. run yarn workspace @acme/two run build - Builds OK

• A branch ready to play with Yarn 2 & PNP that produces this issue
  https://github.com/DimitrK/monorepo-package-manager/tree/pnp/yarn-v2

  1. Switch to branch pnp/yarn-v2 (diff from master)
  2. run yarn set version berry
  3. run yarn install
  4. run yarn workspace @acme/two run build - Throws

Environment if relevant (please complete the following information):

• OS: OSX 10.13
• Node version v13.10.1
• Yarn version: Tested on both 1.22.4 & 2.0.0-rc.29

Additional context
The idea is to move common dev dependencies for all workspaces up in the monorepo root director. That would help us keep consistent, simple and uniform build processes plus dev configuration setup across all of our workspaces.

The following structure gives an approximate view on how package.json files are placed:

.
|
├── apps
|   ├── one
|   |   └── package.json //build script using webpack
|   └── two
|   |   └── package.json 
└── package.json //devDependencies such as webpack

[arcanis]

I've added an FAQ entry explaining how to share scripts without relying on hoisting - does that answer your use case?

https://yarnpkg.com/advanced/qa#how-to-share-scripts-between-workspaces

[DimitrK]

@arcanis I managed to get the hoisting working on that one (see diff).

A new problem presents itself though. Yarn pnp complains about missing dependencies on webpack.config.js found inside the workspace although all the dev dependencies are declared one level above, at root package.json

Error: Your application tried to access html-webpack-plugin, but it isn't declared in your dependencies; this makes the require call ambiguous and unsound.

Required package: html-webpack-plugin (via "html-webpack-plugin")
Required by: /Users/dimitriskyriazopoulos/Development/yarn-scoped-dependency/apps/one/

Require stack:
- /Users/dimitriskyriazopoulos/Development/yarn-scoped-dependency/apps/one/webpack.config.js

Any ideas why is not doing module resolution by looking a directory up until the next available package.json file?

[DanCouper]

Same issue here -- all the dev tooling that I want common to a project (if that tooling relies on plugins) falls over (Eslint, Jest, Webpack etc). Is there any way I can configure this to work as it did on Yarn 1.n? The docs are pretty sparse at the minute and it's difficult to find any information there.

This doesn't seem a strange thing to want (common dev tooling dependencies that are used for everything throughout the project), so I assume I'm missing something obvious here? I mean I can write a script for my project to install everything per-package and ensure everything stays completely in sync, version-wise (or do it manually and then use constraints, or dumping PnP might work, or just downgrade to Yarn 1.n), but the ability to easily define common dependencies in the root is incredibly convenient -- my reason for using a monorepo structure in the first place was in large part to avoid having to do this busywork.

[arcanis]

Have you checked the FAQ entry I referenced in my previous comment?

How to share scripts between workspaces

[DimitrK]

It seems that it changed completely how dependencies get resolved.

I had the impression that even with hoisting disabled on Yarn 1 workspaces, the node module resolution would work as node resolves module dependencies where they keep looking for dependencies on node_modules folder of the closest parent directory with a package.json file that matches the semver. Same on what pnpm does as well.

Instead, now, Yarn 2 seems that keeps each workspace isolated, without looking a directory up if the dependency not found in the workspace's node_modules directory.

In my case at least, aliasing workspace scripts alone won't help and seems counter-productive since doing a require('webpack') within one of the workspace's webpack.config file would still complain that webpack is not a dependency.

Finally, being in a monorepo setup, some devtooling, build and transpile common logic is shared among workspaces. These helper methods sit outside of any workspace at <root>/config file where they import/require many libraries on their own such as toolbelts, babel & webpack plugins in order to function properly. I now have to have of the dev dependencies of this common logic on root package.json and same dev dependnecies plus some additional ones per case sparse on each workspace.

[arcanis]

I had the impression that even with hoisting disabled on Yarn 1 workspaces, the node module resolution would work as node resolves module dependencies where they keep looking for dependencies on node_modules folder of the closest parent directory with a package.json file that matches the semver. Same on what pnpm does as well.

This unfortunately was an incorrect assumption. The exact hoisting rules have always been unspecified (by design), and relying on a specific package being made available simply because it was in a parent workspace has always been incorrect. Should you have added a dependency into your nested workspaces that would have happened to transitively depend on Jest somehow (maybe by mistake!), you'd have required the wrong Jest version.

Instead, now, Yarn 2 seems that keeps each workspace isolated, without looking a directory up if the dependency not found in the workspace's node_modules directory.

In a sense, they always were isolated. The difference is that we now tell you when you break the isolation, which we weren't able to do before (the tech simply didn't exist! 😄).

since doing a require('webpack') within one of the workspace's webpack.config file would still complain that webpack is not a dependency.

Imo the best way to do this would be to add webpack as dev dependency of your various app workspaces, then as a peer dependency of your config-shared workspace.

The drawback is that webpack will then become duplicated across multiple workspaces, but Yarn 2 contains tools that help make sure it scales well - in particular constraints and yarn up. By using them, you can ensure at CI level that a single version is used everywhere.

[DanCouper]

Have you checked the FAQ entry I referenced in my previous comment?

Yes, but that works only (AFAICS) for execution of single binaries -- so the example used is good, as I can put a single version of the TS compiler into the project, then compile using that. It doesn't help with tools that are not standalone (I realise this may be dependent on a particular tool's architecture, how it locates plugin dependencies).

Imo the best way to do this would be to add webpack as dev dependency of your various app workspaces, then as a peer dependency of your config-shared workspace.

The drawback is that webpack will then become duplicated across multiple workspaces, but Yarn 2 contains tools that help make sure it scales well - in particular constraints and yarn up. By using them, you can ensure at CI level that a single version is used everywhere.

This is the issue I want to avoid though. The tooling dependencies are things I only need one of in a project: I don't want multiple versions of Jest or ESlint. The JS ecosystem, with its lack of built-in tooling, causes this issue (and something like Rome, one standalone tool, or Deno with its Go-influenced stdlib seem to be big steps in the correct direction, but they are way off in the distance).

Constraints seem to be most useful (and are something I've wanted for a while), but there is close to zero documentation, and are not something I can reasonably introduce to a team -- I get the basic syntax from using Erlang, and with trial and error can get something functional, but I can't introduce them to a team as things stand (saying go and learn Prolog isn't at all realistic). And up will help a lot, but isn't a replacement.

[arcanis]

Constraints seem to be the correct solution (and are something I've wanted for a while), but there is close to zero documentation, and are not something I can reasonably introduce to a team

The nice thing with constraints is that they aren't something your whole team has to be aware off. The way they are written, it's enough if a single person "owns" their definition. A bit like not everyone has to know all the details of the deploy infrastructure, as long as one person can provide help when needed.

Regarding the documentation bits, there are a few pages I want to write soonish 🙂

[DanCouper]

All the above makes sense given how v2 has been designed. I've been thinking about this a bit more, what my actual problem is here. I think I was expecting everything to work more like Yarn 1, but I appreciate that what I thought was specifically defined behaviour isn't actually in reality defined behaviour, it just works that way almost by accident. The colon script trick is useful. But it seems safer to just copy the big list of devDependencies across to each package.json. My issue is that this is fine when I'm standing the repo up for the first time and it's all nice and clean. As time goes by though, adding new packages that need the exact same large set of dependencies is going to start getting messy (constraints will definitely help, but putting those aside for the minute). It's just that some tooling to do that automatically & avoid human error creeping in would be nice. I dunno, something like a commonDependencies field in the root package.json, and a plugin that read from it & went and automatically added them to each package's devDependencies. Anyway...

👍 on more Constraints documentation, really need to see and play around with a load of examples.

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolution faster). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label.