Berry: [Bug] `yarn workspaces foreach exec` fails

[alubbe]

• [ ] I'd be willing to implement a fix

Describe the bug

yarn workspaces foreach exec fails

To Reproduce

I am trying to run depcheck in every workspace, but it seems that exec, while a valid yarn command, is not passed through by yarn workspaces foreach. Is this a bug or is exec not meant to be supported?

yarn workspaces foreach exec depcheck

Usage Error: Invalid subcommand name for iteration - use the 'run' keyword if you wish to execute a script

[paul-soporan]

yarn exec has been implemented, but it hasn't been released yet. You can do yarn set version from sources to build and use the version from master, which includes yarn exec.

Edit: And yes, it's supported by yarn workspaces foreach, I can confirm that yarn workspaces foreach exec ls works.

[alubbe]

It doesn't work for me - could you reopen the issue?

yarn set version from sources
yarn workspaces foreach exec depcheck

➤ YN0001: Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at B.Writable.write (_stream_writable.js:286:11)
    at e.run (/proj/.yarn/releases/yarn-sources.js:31:458337)

Also, yarn exec is released/working with 2.0.0-rc.33

[paul-soporan]

It doesn't work for me - could you reopen the issue?

Sure, but note that that's a different issue from the original one. Can you try running a simple yarn exec depcheck on one of your workspaces, so we can know if it's an issue with yarn exec or with yarn workspaces foreach?

Also, yarn exec is released/working with 2.0.0-rc.33

Are you... sure? yarn exec was implemented in #1348, which was on May 12, 2020, while 2.0.0-rc.33 was released on May 1, 2020.

$ yarn -v && yarn exec ls
2.0.0-rc.33
Usage Error: Couldn't find a script named "exec".

$ yarn run [--inspect] [--inspect-brk] <scriptName> ...

That's also why you got Invalid subcommand name for iteration when doing yarn workspaces foreach exec depcheck, since the command didn't exist.
Maybe I'm missing something obvious, in which case feel free to correct me.

[alubbe]

You're right, yarn exec is not working on rc.33, I was sure it was - never mind! :)
Anyways, I just re-compiled from sources again, and it still fails:

➤ YN0000: Saving the new release in .yarn/releases/yarn-sources.js
➤ YN0000: Done in 1.88m
➜  x-ray git:(yarn2) ✗ yarn exec depcheck
Internal Error: spawn depcheck ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:267:19)
    at onErrorNT (internal/child_process.js:467:16)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
➜  x-ray git:(yarn2) ✗ yarn workspaces foreach exec depcheck
➤ YN0001: Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at B.Writable.write (_stream_writable.js:286:11)
    at e.run (.yarn/releases/yarn-sources.js:31:458337)
➤ YN0000: Failed with errors in 0.46s

I assume yarn exec depcheck fails because the .bin was not added to the PATH, right? That's fine, I can make that work somehow
But I'm not sure how to handle the yarn workspaces foreach exec depcheck failure

[arcanis]

You're looking for yarn run depcheck, if depcheck is a dependency.

[alubbe]

On the contrary, I'm trying to run it in every workspace, regardless of it being a dependency - precisely because I cannot assume that every developer will remember to add it as a dependency. So I think I want something like yarn workspaces foreach exec yarn depcheck:, assuming depcheck: is defined in one of the workspaces. Is that right?

Any ETA on exec?

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

[merceyz]

@alubbe Could you provide a reproduction please?

[bgotink]

You're looking for yarn run depcheck, if depcheck is a dependency.

@arcanis Yeah, but yarn workspaces foreach run depcheck won't do a thing unless there are depcheck scripts in all workspaces. Is that expected behaviour?
yarn run falls back to installed binaries (or, if the name has a colon, the script of another workspace in the project) but yarn workspaces foreach run doesn't do this.

To actually run depcheck, you could exec run, e.g.

yarn workspaces foreach exec run depcheck

to run the installed depcheck binary, though that'd require every workspace to have a dependency on depcheck. This could be worked around using

yarn workspaces foreach exec run -T depcheck

The distinct different behaviour of yarn workspaces foreach run is a bit confusing imo.

[alubbe]

I think yarn workspaces foreach exec run -T depcheck is the exact thing that I was looking for! But how does this work? Why can exec call run, that's not a valid bash command? And where does -T come from / what does it do? I couldn't find anything here:
https://yarnpkg.com/cli/exec
https://yarnpkg.com/cli/run
https://yarnpkg.com/cli/workspaces/foreach

I've just noticed one more strange thing - I can't quit the process by hitting ctrl+c. I assume that's a bug because I can ctrl+c out of yarn workspaces foreach exec pwd. It seems to be related to exec run.

[bgotink]

I can't quit the process by hitting ctrl+c

I noticed the ^C issue as well yesterday, but I hadn't looked into it yet. Now I have and I've logged it as a bug at https://github.com/yarnpkg/berry/issues/1622.

Why can exec call run, that's not a valid bash command?

When you run yarn exec (or yarn workspaces foreach exec) yarn adds a folder to the path with a couple of scripts in it:

• node runs the node.js version that yarn was run with (so if you run path/to/node .yarn/releases/yarn.cjs exec node you'll get the same node version.
• yarn and yarnpkg to run the yarn version you're executing, not the global yarn
• run as alias to yarn run

And where does -T come from / what does it do?

The -T (or --top-level) switch is apparently a hidden option. I'm not sure why.
The option tells yarn to look for the binary to run in the root workspace, not in whatever workspace yarn is running. This allows for using tools installed in the root workspace in every workspace of your project.

[arcanis]

run as alias to yarn run

That's funny, I never considered it would work with exec. I initially made it this way because it turns out there's no real portable way to run a script regardless of the package manager (each time you need to explicitly specify npm / yarn / pnpm, ...). I haven't advocated it to other package managers though, so it's not portable either (yet).

[arcanis]

The -T (or --top-level) switch is apparently a hidden option. I'm not sure why.

I only used it for the node-gyp hack (when a package doesn't properly list it in their dep, we still need to run it from somewhere), and I wasn't sure whether it would make sense to advocate it as an official feature or rather an implementation detail...

[arcanis]

Yeah, but yarn workspaces foreach run depcheck won't do a thing unless there are depcheck scripts in all workspaces. Is that expected behaviour?

It will skip over scripts that don't declare a script of this name, yes. However it probably should execute it too if there's a binary of the specified name, so if it doesn't do so already I'd see that as a bug.

[bgotink]

The -T (or --top-level) switch is apparently a hidden option. I'm not sure why.

I only used it for the node-gyp hack (when a package doesn't properly list it in their dep, we still need to run it from somewhere), and I wasn't sure whether it would make sense to advocate it as an official feature or rather an implementation detail...

For tools like these I actually see this as a valid usecase. I consider depcheck a _project_ tool that you want to run in every workspace. Having to install it in every workspace feels icky, especially if the tool is only ever run in the entire project as a whole.
The same goes for the angular CLI imo, which is a project tool (it's configured in one central angular.json file for the entire project) that you might want to run in a certain workspace's scripts.

For most tools this won't apply, but tools like eslint, prettier, @angular/cli, depcheck I do like it.

[DerekZiemba]

I straight up cannot yarn workspaces foreach to run at all.
I can't do yard workspaces foreach run test , but normal yarn workspaces run test works fine.
The goal is to run the build command , but not every workspace needs building so some lack the script.

Stuff I've tried that fail with the same error:

• Basic sanity checking stuff like yarn workspaces foreach exec ls
• Running commands in different consoles such as vscodes's embedded, ConEmu, Git Bash, and Powershell
• Forked the following 3 example repos, and still had no success:
  lerna-yarn-workspaces-example, typescript-yarn-workspace-example, yarn-workspaces-simple-monorepo

Am I doing something wrong here? I'm currently just testing yarn and this feature to assess if it's a better solution than npm and npm link. So far it doesn't appear so.

$ yarn workspaces info
yarn workspaces v1.22.4
{
  "example-1": {
    "location": "examples/example-1",
    "workspaceDependencies": [],
    "mismatchedWorkspaceDependencies": []
  },
  "example-2": {
    "location": "examples/example-2",
    "workspaceDependencies": [
      "example-1"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "other-example": {
    "location": "other-example",
    "workspaceDependencies": [
      "example-1",
      "example-2"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "@test/common": {
    "location": "packages/common",
    "workspaceDependencies": [
      "example-1"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "@test/server": {
    "location": "packages/server",
    "workspaceDependencies": [
      "@test/common",
      "example-1",
      "example-2",
      "other-example"
    ],
    "mismatchedWorkspaceDependencies": []
  }
}

[paul-soporan]

@DerekZiemba You're using Yarn Classic (v1), while yarn workpaces foreach is a Yarn 2 feature. You can consult the installation guide to install Yarn 2. If it still doesn't work, you should probably open another issue, as it's unlikely it would be related to this one.

[DerekZiemba]

Oh, I had no idea. Thought it was latest stable. https://classic.yarnpkg.com/en/docs/install/#windows-stable is the first Google result when searching yarn windows
My bad, I'll give that a try.

[DerekZiemba]

@paul-soporan So that migration was super painful and way more complicated than expected. Especially for a simple sample repo.

Anyway I think I've now been stopped by an actual bug regarding the dependency graph. This same dependency graph was fine in yarn classic. Also there is a MaxListenersExceededWarning.
If anyone would like to take a look at the code and try it themselves:
https://github.com/DerekZiemba/yarn-V2-workspaces-simple-monorepo
Let me know if I should start a new issue.

bash $ yarn workspaces foreach -t run build (node:3408) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit ➤ YN0000: src/common.ts:2:22 - error TS2307: Cannot find module 'example-1' or its corresponding type declarations. ➤ YN0000: ➤ YN0000: 2 import { echo } from 'example-1'; ➤ YN0000: ~~~~~~~~~~~ ➤ YN0000: ➤ YN0000: ➤ YN0000: Found 1 error. ➤ YN0000: ➤ YN0000: The command failed for workspaces that are depended upon by other workspaces; can't satisfy the dependency graph ➤ YN0000: Failed with errors in 1.55s

[paul-soporan]

@DerekZiemba I've left you feedback here: https://github.com/DerekZiemba/yarn-V2-workspaces-simple-monorepo/issues/1
We can continue the discussion there.