Berry: [Feature] Removing .yarn folder should yield an understandable error

[scarlac]

• [ ] I'd be willing to implement this feature
• [ ] This feature can already be implemented through a plugin

Describe the user story

Removing node_modules folder is NodeJS equivalent of "have you tried restarting your pc"?
I daily see this recommended in discussions, on forums, in offices environments.

So if a user tries to do the equivalent of:

rm -rf node_modules

The natural assumption would be:

rm -rf .yarn

Which fully screws up their entire project, including yarn, putting them into an unusable state:

18:48:38 ~/projects/myproject ⚡️ yarn
internal/modules/cjs/loader.js:1083
  throw err;
  ^

Error: Cannot find module '/Users/scarlac/projects/myproject/.yarn/releases/yarn-2.1.1.cjs'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:1080:15)
    at Function.Module._load (internal/modules/cjs/loader.js:923:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

The user now has normal way to recover, and they have to resort to git or backups. This is a terrible experience, and it's often not an issue beyond configuration to remove dot-folders with other tools. With Yarn 2 the story is now that it's fatal. There is no clear yarn command that leads to recovery (yarn doctor will cause the same error).

Describe the solution you'd like

The issue is that the binary is shipped with the configuration folder. Given this design decision, the yarn command should either:

1. Still function
2. Display a human-readable error, saying why it won't function
3. Display a human-readable error, saying how the human can fix the issue
4. Fix itself and display a warning

Currently, we are bleeding the out onto the screen, showing internal javascript.

Examples of error message that may be helpful:

• Yarn binary not found (.yarn/releases/yarn-2.1.1.cjs). Did you remove it?
• Yarn2 needs the .yarn folder to function. Please run XXXX to generate it from scratch
• Local yarn binary not found. .yarn/ folder may not be removed like node_modules/. Run XXXX to generate it

Describe the drawbacks of your solution

Increases code complexity slightly.

Describe alternatives you've considered

Automatically re-generating the .yarn folder from scratch with the latest binary would be ideal but I am not sure you guys agree (as I imagine the decision to put the binary next to the cache was an elaborate informed choice and removal of .yarn was considered to be 'wontfix' regardless).

[paul-soporan]

Removing node_modules folder is NodeJS equivalent of "have you tried restarting your pc"?
I daily see this recommended in discussions, on forums, in offices environments.

This isn't recommended with Yarn 2. The node-modules linker is much more stable, which means that bugs that require deleting the node_modules folder are very rare.

The natural assumption would be:
rm -rf .yarn

We've made it clear that the .yarn/ folder contains the Yarn 2 binary + plugins. Why would anyone blindly rm -rf the entire .yarn/ folder? Did you actually want to only clear the cache? (rm -rf .yarn/cache aka yarn cache clean) I don't see any reason why anybody would want to remove absolutely everything.

The issue is that the binary is shipped with the configuration folder. Given this design decision, the yarn command should either:
1) Still function
2) Display a human-readable error, saying why it won't function
3) Display a human-readable error, saying how the human can fix the issue
4) Fix itself and display a warning

1) It can't, you've deleted the Yarn binary... We can't have Yarn 1 invoke the ghost of Yarn 2.
2) I agree that the error can be better. The improvement would have to be made in Yarn Classic, as with the current installation strategy Yarn Classic tries to spawn the Yarn 2 binary you've referenced via .yarnrc.yml and fails. Personally I don't find the current error message that unreadable. It's just a Node MODULE_NOT_FOUND error. ¯\_(ツ)_/¯
3) You've deleted the Yarn 2 binary. This means that your options are:
a) restore it from a backup / git
b) run yarn set version berry yet again, but this isn't a good solution, as it can modify your cache archive checksums and your lockfile, missing the entire point of per-project installs.
4) It can't fix itself, you killed it. Yarn 1 has no idea which Yarn 2 binary was used before. It can't just guess, as that guess will affect everything going forward. ¯\_(ツ)_/¯

Automatically re-generating the .yarn folder from scratch with the latest binary would be ideal but I am not sure you guys agree (as I imagine the decision to put the binary next to the cache was an elaborate informed choice and removal of .yarn was considered to be 'wontfix' regardless).

Yep, that's definitely a won't fix. Even if we found a way to somehow magically regenerate the Yarn binary, we'd have absolutely no way to regenerate plugins (which also yield a MODULE_NOT_FOUND error when not found).

So, in the end, the only actionable part I see is improving the error message.

[arcanis]

One problem with improving the error message is that it requires to hotfix Yarn 1, which is a huge pain - the tests don't even pass there anymore for whatever flaky reason 😞 So it would be "nice to have", but I'd postpone that until after we have pmm, which should shake things a bit in this regard.

[rockerBOO]

Ok I just found myself in this same situation. Maybe we can guide people into the right solutions at the "problem" cases. Like @scarlac has stated, clearing the build artifacts in other node apps can usually be rebuilt without worrying too much. In this case we have a .yarnrc.yml file which defines our configuration so I am aware that this file should stay.

In the documentation there is Zero-Installs. This suggests we will be able to commit our dependencies and not have to install them in the build steps. Going from this idea the goal was to use yarn2 to keep my dependencies in check. We also get a .pnp.cjs/.pnp.js file as well. So from package.json,yarn.lock,.yarnrc.yml,and .pnp.cjs I would've thought we could effectively rebuild the yarn cache.

Now in my particular situation. I am maybe trying to do too many new things at the same time, but I got to a point where I couldn't figure out how to solve it. Could be a project specific issue but this is where I couldn't figure out how to solve it.

cross-env NODE_ENV=development BROWSERSLIST_ENV=development webpack-dev-server -d --mode development"

$ yarn dev
/mnt/c/projects/***/assets/.pnp.cjs:44598
    throw firstError;
    ^

Error: Qualified path resolution failed - none of the candidates can be found on the disk.

Source path: /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-cli-virtual-f4bc12778f/6/home/rockerboo/.yarn/cache/webpack-cli-npm-4.0.0-beta.8-639c7f60d2-f012b213b4.zip/node_modules/webpack-cli/bin/config-yargs
Rejected candidate: /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-cli-virtual-f4bc12778f/6/home/rockerboo/.yarn/cache/webpack-cli-npm-4.0.0-beta.8-639c7f60d2-f012b213b4.zip/node_modules/webpack-cli/bin/config-yargs
Rejected candidate: /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-cli-virtual-f4bc12778f/6/home/rockerboo/.yarn/cache/webpack-cli-npm-4.0.0-beta.8-639c7f60d2-f012b213b4.zip/node_modules/webpack-cli/bin/config-yargs.js
Rejected candidate: /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-cli-virtual-f4bc12778f/6/home/rockerboo/.yarn/cache/webpack-cli-npm-4.0.0-beta.8-639c7f60d2-f012b213b4.zip/node_modules/webpack-cli/bin/config-yargs.json
Rejected candidate: /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-cli-virtual-f4bc12778f/6/home/rockerboo/.yarn/cache/webpack-cli-npm-4.0.0-beta.8-639c7f60d2-f012b213b4.zip/node_modules/webpack-cli/bin/config-yargs.node

Require stack:
- /mnt/c/projects/***/assets/.yarn/$$virtual/webpack-dev-server-virtual-5dc10466bd/6/home/rockerboo/.yarn/cache/webpack-dev-server-npm-3.11.0-42b1303dbc-1d34457456.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js
    at internalTools_makeError (/mnt/c/projects/***/assets/.pnp.cjs:44343:34)
    at resolveUnqualified (/mnt/c/projects/***/assets/.pnp.cjs:45344:13)
    at resolveRequest (/mnt/c/projects/***/assets/.pnp.cjs:45368:14)
    at Object.resolveRequest (/mnt/c/projects/***/assets/.pnp.cjs:45428:26)
    at Function.external_module_.Module._resolveFilename (/mnt/c/projects/***/assets/.pnp.cjs:44575:34)
    at Function.external_module_.Module._load (/mnt/c/projects/***/assets/.pnp.cjs:44439:48)
    at Module.require (internal/modules/cjs/loader.js:1140:19)
    at require (internal/modules/cjs/helpers.js:75:18)
    at Object.<anonymous> (/mnt/c/projects/***/assets/.yarn/$$virtual/webpack-dev-server-virtual-5dc10466bd/6/home/rockerboo/.yarn/cache/webpack-dev-server-npm-3.11.0-42b1303dbc-1d34457456.zip/node_modules/webpack-dev-server/bin/webpack-dev-server.js:65:1)
    at Module._compile (internal/modules/cjs/loader.js:1251:30)

Attempts at using yarn cache clear did not solve this problem.

Specifically this line made me think that there was some cache issue. Note that I am changing packages around almost constantly. I am not sure how I can resolve this error.

Error: Qualified path resolution failed - none of the candidates can be found on the disk.

So to "fix" whatever weird state my package was in, and to rebuild from my package.json/yarn.lock files, I thought I would clear the .yarn directory. After clearing my .yarn directory I ran yarn.

$ rm -R .yarn
$ yarn
➤ YN0000: ┌ Resolution step
***
➤ YN0000: └ Completed in 1.02s
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 1.48s
➤ YN0000: ┌ Link step
➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.1.3#builtin<compat/fsevents>::version=2.1.3&hash=495457 The platform linux is incompatible with this module, linking skipped.
➤ YN0062: │ fsevents@patch:fsevents@npm%3A1.2.13#builtin<compat/fsevents>::version=1.2.13&hash=495457 The platform linux is incompatible with this module, linking skipped.
➤ YN0001: │ Error: Assertion failed: Expected the package to have been registered
    at a (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:132408)
    at R (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:132561)
    at M (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:131091)
    at $.finalizeInstallWithPnp (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:148094)
    at async $.finalizeInstall (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:210036)
    at async ne.linkEverything (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:299220)
    at async /mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:305386
    at async f.startTimerPromise (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:313773)
    at async ne.install (/mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:305331)
    at async /mnt/c/projects/***/.yarn/releases/yarn-berry.js:2:48190
➤ YN0000: └ Completed in 5.39s
➤ YN0000: Failed with errors in 8.5s

My attempts at solving this was to re-install yarn. And then I came to this repo to see if it was something fixable.

---

Maybe it's misleading to have .yarn have "cache" and required files, when other files outside of .yarn are also required. Having to resort to rm is not ideal but if I am being blocked and time is limited, I will just do it. I can go back to git if required.

In this case the problems leading up to and after the error occured did not indicate what actually was going on to me. I still don't know what the original error means or how to solve it. Of course documentation may highlight solutions (maybe they exist already).

I spent around 10 hours trying to upgrade to yarn v2 to support Zero-Install, with a number of packages not working properly, but eventually I got to a broken yarn version with no way to resolve. The time isn't an issue for me but I did invest some time trying to figure it out and still got to a dead-end.

[merceyz]

The issue you have with webpack is this issue: https://github.com/webpack/webpack-dev-server/issues/2424, you're using a incompatible version of webpack with webpack-dev-server

[scarlac]

As @rockerBOO points out, although for separate reasons, having a .yarnrc.yml and a separate .yarn folder implies that .yarn can be removed (should .yarnrc.yml be moved into .yarn/ in Yarn3 perhaps?). I can wipe most dot files and all I lose is configuration (e.g. .eslintrc, .git/, .npmrc, .env, .vscode). In all of these situations they are re-generated, a readable error occurs like "Unable to read configuration from XXX", or often they just fall back to defaults of the tool. Even git is helpful enough to say:

⚡️ git log
fatal: not a git repository (or any of the parent directories): .git

Instead Yarn2 crashes. It makes Yarn2 harder to use than it has to be. I'd really just like us to all use tools that we don't have to read the manual to use.

If Yarn Classic is not going away couldn't we update it to handle this situation, given it already reads .yarnrc.yml? Should I move this suggestion to different repo? I really just want Yarn2 to get wide adoption which requires a smooth experience.

[FallingSnow]

I've been running into the Error: Assertion failed: Expected the package to have been registered issue as well. Can't seem to get around it while using nodeLinker: node-modules.

[casperan]

Maybe it would be nice for the yarn user with an implementation where everything can be recovered from a simple package.json, that yarn can process and use to rebuild .yarn/ with bin, cache, yarn.yml, yarn.lock, pnp, etc...

Maybe this could help ease the cognitive switch between npm and yarn, and thus reduce the fear of trying a switch to the better solution at any given time.

As a yarn user my biggest fear is that .yarn becomes like Meteor was an maybe still is to some extent, limiting and restricting users from the general javascript ecosystem, and getting a feel of being locked in, and constrained with a rigid and counterproductive less free and compatible system.

[arcanis]

Just to be clear: the .yarn folder is where we store the artifacts. The cache, but also the release. That Yarn crashes if you remove the release that's configured in your .yarnrc.yml is expected. We could have a slightly better error message, but frankly the current "File not found" doesn't seem very ambiguous as to what the problem is, and I don't think it's pressing to fix that, especially since it requires making a release on the v1 branch, which requires a very good reason.

Apart from the artifacts referenced in your very configuration (release, plugins), everything else in this folder can be safely removed, and Yarn will just regenerate it on subsequent installs.

[arcanis]

Finally, note that if you check-in the binaries as we suggest, the semantic way to "reset" your .yarn folder is:

rm -rf .yarn && git checkout .yarn && yarn

[scarlac]

@arcanis Provided that you use git and that you have it set up in git, which I did not have as I was trying to migrate.
The file not found can be figured out if one bothers to read the entire thing but the solution isn't obvious.
I am merely asking for an error message that is solution oriented kinda like ~"This is what went wrong. Here's how to fix it".

Yarn2 is already a hard sell given the defaults. I'd just like users to be able to not have to understand the internals to recover from a fairly easily encountered situation.

I'll leave it at that. If you feel it's a wontfix, feel free to mark it as such.

[lehni]

I'm experiencing the same problem as @FallingSnow: With nodeLinker: node-modules, I keep getting this error, and even resetting .yarn does not seem to help

[merceyz]

@lehni Unrelated to the topic of this issue, please open a seperate issue with a reproduction

[cdestiawan]

@paul-soporan

We've made it clear that the .yarn/ folder contains the Yarn 2 binary + plugins. Why would anyone blindly rm -rf the entire .yarn/ folder? Did you actually want to only clear the cache? (rm -rf .yarn/cache aka yarn cache clean) I don't see any reason why anybody would want to remove absolutely everything.

Yes, I was one of them, and now I can't even use yarn 2 (and even yarn 1) anymore. Don't know how to use it again. Yarn should have a way to recover itself, in case if there are some stupid user (like me) removing .yarn folder by accident. And now I switch to pnpm, instead of yarn.

[devoto13]

@cdestiawan I assume yarn set version berry will restore the executable if it was deleted by accident.

[merceyz]

I'll close this as there is nothing we can do about it from Yarn V2's side.

---

Until it has been released with Node (if it gets accepted) you can manually enable https://github.com/nodejs/corepack to handle this.

npm uninstall yarn -g
npm install corepack -g

Add this to your package.json instead of adding the binary locally

  "packageManager": "[email protected]",