Esbuild: Yarn PnP support

[rtsao]

Have you considered support for Yarn PnP? PnP is a faster alternative to node_modules used by Yarn 2. Webpack, Parcel, Rollup, and TypeScript currently support PnP.

Because esbuild is implemented in Go rather than JS, PnP support will probably be a larger undertaking, but I figured I would create an issue to gauge maintainer and community interest.

[evanw]

No I haven't. As you pointed out, it's not a good fit for esbuild because the module resolution involves executing arbitrary JavaScript code instead of following a well-defined algorithm.

I'm not sure if it's a valid alternative, but you could consider pnpm instead. That uses symlinks as a faster alternative to node_modules and esbuild has been updated to support it: #67.

It's possible that esbuild might have some form of "resolve plugin" API in the future which would make this possible. Either that or it could potentially expose an API that would let you build your own bundler that supports PnP and uses esbuild as a backend. But such an API won't be coming soon because esbuild isn't that mature yet.

[rtsao]

Theoretically speaking, if someone was willing to implement PnP resolution natively in Go (perhaps as a separate library), would you consider accepting PnP resolution into esbuild core (perhaps as a CLI flag)?

I'm wondering what approach might incur the least amount of maintenance burden. It might be worth just implementing PnP in Go rather than dealing with JS FFI or having to maintain some plugin abstraction. I haven't had a chance to dive into the codebase yet, but do you think the current resolution code could support an alternate resolution algorithm without too much extra complexity?

Obviously PnP is relatively new to the ecosystem, but I could see PnP support be a blocker for a growing number users. That being said, it is totally understandable if you'd be unwilling to maintain PnP support.

[evanw]

It doesn't seem appropriate to have this be a part of esbuild, so I don't think I would accept a PR for this. After reading https://github.com/yarnpkg/yarn/issues/6388 it looks like they deliberately made the resolution strategy depend on executing JavaScript because they want to maintain the ability to evolve the implementation. So it seems like the intended design is for the loader to call into their library.

A JavaScript plugin would be the most appropriate way to do this. Unfortunately esbuild isn't ready for JavaScript plugins. I plan to start experimenting with them at some point, and if/when I do I will keep a "resolve plugin" API in mind for this use case.

[arcanis]

Hey 👋 A few tidbits of context I shared on Discord:

the module resolution involves executing arbitrary JavaScript code instead of following a well-defined algorithm

It is well-defined; the problems are 1/ the data structures, which we don't generate in a stable format (but you can easily treat the generated PnP API as an intermediary to a stable format of your choice, for example a stable JSON file or import maps), 2/ the algorithm itself, which you need to reimplement.

Since there are very few projects that need to implement the PnP resolution in other languages, we didn't see the need to provide the runtime in other languages (it would come with maintainance work that we don't really need).

That being said, note that the .pnp.js file is also a valid standalone JS script ... if you run it without arguments, it'll spawn in daemon mode and you'll be able to interact with it through stdin/stdout there's a very basic protocol that allows calling the resolution and reading back the results that's how we implemented Flow PnP support, back in the day. Caveats: you still need to execute JS, in the end / resolution is a bit slower / you still need to add the zip layer support to the native tool (this is what was the final blocker with Flow).

[evanw]

Thanks for stopping by to provide more information.

it'll spawn in daemon mode and you'll be able to interact with it through stdin/stdout

This sounds like what I had in mind for JavaScript plugins (see #111). So that confirms to me that this would probably be best as a JavaScript plugin, since it makes sense to build a mechanism like that into esbuild once, not twice. The plugin could just call out to Yarn's API.