Berry: [Bug] Couldn't find the PnP package map

[jeffrson]

Being curious, I just tried to get started from yarn 1.13.0:

yarn policies set-version nightly
yarn policies set-version berry

then
yarn init

Unfortunately I get this error message:

C:\temp\yarn_berry>yarn
Error: Couldn't find the PnP package map at the root of the project - run an install to generate it
at PnpLinker.findPackageLocator (C:\temp\yarn_berry.yarn\releases\yarn-berry.js:115987:19)
at Project.findLocatorForLocation (C:\temp\yarn_berry.yarn\releases\yarn-berry.js:33834:42)
at Function.find (C:\temp\yarn_berry.yarn\releases\yarn-berry.js:33667:39)

yarn install, yarn add typescript - same error.

Here's the output of yarn config -v - there appear to be some strange paths in it:

C:\temp\yarn_berry>yarn config -v
bstatePath Path of the file where the current state of the built packages must be stored 'C:\temp\yarn_berry/.yarn/build-state.yml'
cacheFolder Folder where the cache files must be written 'C:\temp\yarn_berry/.yarn/cache'
defaultLanguageName Default language mode that should be used when a package doesn't offer any insight 'node'
defaultProtocol Default resolution protocol used when resolving pure semver and tag ranges 'npm:'
enableColors If true, the CLI is allowed to use colors in its output true
enableNetwork If false, the package manager will refuse to use the network if required to true
enableScripts If true, packages are allowed to have install scripts by default true
enableTimers If true, the CLI is allowed to print the time spent executing commands true
frozenInstalls If true, prevents the install command from modifying the lockfile false
globalFolder Folder where are stored the system-wide settings 'C:\temp\yarn_berry/C:\temp\yarn_berry/C:\Users\username\AppData\Local'
httpProxy URL of the http proxy that must be used for outgoing http requests null
httpsProxy URL of the http proxy that must be used for outgoing https requests null
ignorePath If true, the local executable will be ignored when using the global one true
initLicense License used when creating packages via the init command null
initScope Scope used when creating packages via the init command null
initVersion Version used when creating packages via the init command null
lastUpdateCheck Last timestamp we checked whether new Yarn versions were available '1552409146419'
lockfilePath Path of the file where the dependency tree must be stored 'C:\temp\yarn_berry/yarn.lock'
npmRegistryServer URL of the selected npm registry (note: npm enterprise isn't supported) 'https://registry.yarnpkg.com'
pnpDataPath Path of the file where the PnP data (used by the loader) must be written 'C:\temp\yarn_berry/.pnp.data.json'
pnpEnableInlining If true, the PnP data will be inlined along with the generated loader true
pnpIgnorePattern Regex defining a pattern of files that should use the classic resolution null
pnpPath Path of the file where the PnP loader must be written 'C:\temp\yarn_berry/.pnp.js'
pnpShebang String to prepend to the generated PnP script '#!/usr/bin/env node'
pnpUnpluggedFolder Folder where the unplugged packages must be stored 'C:\temp\yarn_berry/.yarn/unplugged'
preferInteractive If true, the CLI will automatically use the interactive mode when called from a TTY false
rcFilename Name of the files where the configuration can be found '.yarnrc'
virtualFolder Folder where the symlinks generated for virtual packages must be written 'C:\temp\yarn_berry/.yarn/virtual'
yarnPath Path to the local executable that must be used over the global one 'C:\temp\yarn_berry/C:\temp\yarn_berry/.yarn\releases\yarn-berry.js'

Environment if relevant (please complete the following information):

• OS: Windows (you might have guessed it)
• Node version: 10.15.3
• Yarn version: v2.0.0

[arcanis]

I'm trying something different in the v2 regarding portability: instead of using either path.win32 or path.posix depending on the system, all primitives use path.posix (basically Windows path would be handled as if they were /mnt/c/foo instead of C:\foo). Then it's within our FS layer that the translation is done, right before actually accessing the files.

Now the problem is that I haven't yet implemented this translation, so things are a bit wonky at the moment (hence the weirdly concatenated paths) 😛 would you be interested to give a look at how hard it would be to implement it?

[Embraser01]

Hey !
I was also trying to use Berry on a Yarn v1 project to see how it would go before I stumbled on this...

I'd be happy to help!

I'm trying to understand the code for now, nice work ! It's quite easy to understand 😅

So in order to get it working on Windows, we (theoretically) just need to complete these 2 functions ?

Also, all FS functions does not seems to use these 2 functions, like here:

https://github.com/yarnpkg/berry/blob/8a7b0cce2110e00cd652cfeee6333ddc7b1b8b70/packages/berry-fslib/sources/NodeFS.ts#L209-L213

Is there a specific reason for this ?

Again, nice work 🎉

[arcanis]

Awesome! Thanks for your help 😃

So in order to get it working on Windows, we (theoretically) just need to complete these 2 functions ?

It'll be a very good start, but there will be a small catch with API boundaries:

• First, the functions will also have to become static instead of member functions for the reason described in the two next points. Not a big deal!

• The CLI currently gets its cwd from Concierge (the CLI framework we use), which currently uses process.cwd(). In order to work properly the path should be "portable-ified" before being sent to the CLI. I believe it should be done here, by doing something like this:

  concierge.runExit(`yarn`, process.argv.slice(2), {
  cwd: NodeFS.toPortablePath(process.cwd()),
  });
  

• It goes the other way around for things like the child_process API - which require a valid system path when specifying the process cwd. We mostly use it here, which should be something like:

  cwd: NodeFS.fromPortablePath(state.cwd),
  

There might be another place or two that have similar subtleties, but I think it should be mostly it - and hopefully will be covered by our tests :)

[Embraser01]

After testing on my machine, it build the bundle but give errors for plugin-hub requiring @berry/ui.

Here is the logs:

The following plugins will be compiled in the final bundle:

- @berry/plugin-essentials
- @berry/plugin-init
- @berry/plugin-constraints
- @berry/plugin-file
- @berry/plugin-github
- @berry/plugin-http
- @berry/plugin-link
- @berry/plugin-npm
- @berry/plugin-pnp

ERROR in N:\berry\packages\plugin-hub\sources\components\EditModeItem.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\components\EditModeItem.tsx(4,57)
      TS2307: Cannot find module '@berry/ui'.

ERROR in N:\berry\packages\plugin-hub\sources\components\RequestBar.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\components\RequestBar.tsx(4,43)
      TS2307: Cannot find module '@berry/ui'.

ERROR in N:\berry\packages\plugin-hub\sources\components\TopBar.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\components\TopBar.tsx(3,43)
      TS2307: Cannot find module '@berry/ui'.

ERROR in N:\berry\packages\plugin-hub\sources\components\WorkspaceTree.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\components\WorkspaceTree.tsx(5,43)
      TS2307: Cannot find module '@berry/ui'.

ERROR in N:\berry\packages\plugin-hub\sources\screens\WorkspacesScreen.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\screens\WorkspacesScreen.tsx(5,51)
      TS2307: Cannot find module '@berry/ui'.

ERROR in N:\berry\packages\plugin-hub\sources\screens\workspaces\WorkspaceView.tsx
[tsl] ERROR in N:\berry\packages\plugin-hub\sources\screens\workspaces\WorkspaceView.tsx(5,68)
      TS2307: Cannot find module '@berry/ui'.

I don't understand why the ts-loader fails at requiring the @berry/ui package or why it tries to build @berry/plugin-hub (it's not in the plugin list).

I don't really know how to debug this because I can't really edit dependencies inside node_modules 😜, any tips to debug this?

[arcanis]

I don't understand why the ts-loader fails at requiring the @berry/ui package or why it tries to build @berry/plugin-hub (it's not in the plugin list).

Typescript typechecks the whole codebase every time, even if they don't get used in the generated bundle.

I don't really know how to debug this because I can't really edit dependencies inside node_modules 😜, any tips to debug this?

I usually debug in two different ways:

• Either I edit the .pnp.js file to put some console.log within resolveToUnqualified (it's even easier now that we've added the wrapper function, since you can just add a try/catch statement to catch & print & rethrow the error).

• Or I actually edit files within the zip archives 🙂 Emacs on Linux/OSX supports read/write access to files stored within zip archives, which helps a lot. I'm not sure how to do that on Windows - the vscode-zipfs package in the repository is a prototype extension for VSCode that plugs ZipFS within VSCode, but it's not published to the store or anything yet 🤔

[Embraser01]

It was actually a weird bug with the cache and symlinks (I think), after reseting the .yarn/ folder (delete and git reset --hard) the yarn build:cli works! :tada:

I can also confirm that yarn install is stable between Windows and Linux (WSL). The only things to note are:

• You must enable symlinks in git with git config core.symlinks true to handle virtuals
• You must enable autocrlf in git with git config core.autocrlf true otherwise all package.json will be detected as changed

[arcanis]

You must enable symlinks in git with git config core.symlinks true to handle virtuals

I wonder if this could somehow be detected when the configuration isn't quite right? For example in VirtualFetcher we could detect whether the symlink is actually stat'd as a file and throw an report error to inform of the problem (and suggest the commands to fix it, if needed). It looks like the kind of things that could cause frustration otherwise 🤔

You must enable autocrlf in git with git config core.autocrlf true otherwise all package.json will be detected as changed

Yep I don't fully understand this one, but from what I read here it seems to be a big mess so I guess that's fine and will be to the users to figure out what they want to do.

[arcanis]

I think we can close this issue since the initial problem was fixed. There's still a bit more work needed for Windows to pass our CI, but it should be inconsequential for most users 🎉

[Embraser01]

I wonder if this could somehow be detected when the configuration isn't quite right? For example in VirtualFetcher we could detect whether the symlink is actually stat'd as a file and throw an report error to inform of the problem (and suggest the commands to fix it, if needed). It looks like the kind of things that could cause frustration otherwise 🤔

👍 So, if we find a file in virtual that isn't a symlink, then throw an error with the suggestion?
Also, this problem could happen in Linux with WSL (if the project is cloned from Windows and then used in WSL, it would also fail).

Yep I don't fully understand this one, but from what I read here it seems to be a big mess so I guess that's fine and will be to the users to figure out what they want to do.

I agree, it's something I've come across multiple times so I'm used to it 😃

[arcanis]

Also, this problem could happen in Linux with WSL (if the project is cloned from Windows and then used in WSL, it would also fail).

Yep, we can check this with os.release() (or just is-wsl) to point out to different messages in the two cases 👍

[BruceHubbard]

I still had this problem and it turned out it was because I had an unrelated package.json in a folder above my project. I ran yarn build from the /Users/bruce/code/myproject folder on my machine and I had a package.json sitting in Users/bruce/code which was not in the project and unrelated to the project but yarn saw that as an ancestor and used that for my lockfilePath. Once I removed that package.json file I got past this error. The code that does this is in getRcData in Configuration. It keeps traversing ancestor directories and keeping track of the last package.json it finds. Hope that helps someone.

[marco-silva0000]

My team also went into this issue on a machine that had a stray package.json as well.

Maybe the error message could be clearer

[arcanis]

I've opened https://github.com/yarnpkg/berry/pull/294 to improve that 🙂