Bit: Dependency resolver extension

[ranm8]

Dependency resolver will be responsible for build-time resolution of component dependencies. It will support the package managers: npm, librarian, pnpm, yarn (if easy win).

• Workspace resolution - resolve each component as a workspace and dedup.
• Capsule resolution -> putting real Semver instead of the used file protocol and rebuild links after dependency resolution.
• Check for best practices for cross-platform symlinks. Talk to @davidfirst before.
• Check how workspace resolution can be done programmatically (e.g. package.json).

[imsnif]

Hi, a few notes regarding workspace resolution:

The Problem

We would like our workspace to be clean, include only a "root" package.json and include all the components without their package.jsons. We would like to have a populated node_modules folder at the root of the workspace so that IDEs and other analytical tools can read from it and use it for type completions and the like.

We have the dependency information for the various components in bit, but we would like to avoid having to write them to the HD even temporarily in order to run a package manager on them (with workspaces or without).

Possible solutions

Use npm programmatically

The official line from npm (for several years now) has been this: https://github.com/npm/npm/issues/8216#issuecomment-100397681
If we decide to do this anyway, we would have to either:

1. install npm as our dependency for a specific version that we know will work (which will bloat our app considerably)
2. rely on the user's globally installed npm (which will force us to keep track of the working versions including breaking changes that can happen randomly in patch versions).

Use pnpm programmatically

This is actually an ok solution. We would have to use it as a dependency, which will still bloat our app somewhat (I can find out the exact numbers), but we can mostly rely on the @pnpm/supi package, which performs the installation itself. How well this works with workspaces (which we will need in order to install multiple components) is anyone's guess, but I think it's worth a try.

The main drawback with this approach is that pnpm is very opinionated about how the node_modules folder should look. Their workaround of using the --shamefully-flatten flag (which we will likely have to have on by default) does not work 100% of the time. The solution they give in their issues is often "the dependency package is at fault, they should not require modules they do not directly depend on". This, while true and understandable, is not helpful in our situation - and I believe does not scale to what we plan on doing.

Here are some example open issues, but there are a lot more in their repo:
https://github.com/pnpm/pnpm/issues/1262
https://github.com/pnpm/pnpm/issues/1592
https://github.com/pnpm/pnpm/issues/1257
https://github.com/pnpm/pnpm/issues/1055
https://github.com/pnpm/pnpm/issues/991

Using our own package manager, Librarian, to dump the node_modules to the HD

Librarian already has most of the capabilities we need, and bringing it the rest of the way is not a terribly long road (likely not much longer than getting pnpm to work the way we want it to). We would simply be instructing it to dump packages to the HD the way we want them rather than keeping them in memory.

We fully maintain it and can quickly address issues that come up. In addition, it has been battle tested in the playground for almost a year now and we've had very few issues with it (most of the issues were cache/infrastructure issues and not issues coming from the resolution/hoisting itself).

[imsnif]

After discussing this with @ranm8 and @qballer, we reached the following conclusion:

By default:

• We're going to use librarian to manage the user workspace (one created with bit new), by dumping the node_modules to the HD (like npm/yarn).
• In the capsules, we'll be using librarian in "fs-monkey-patch" mode.

The capsules would also be able to work in "external package manager" mode. This mode can either be configured by the user (by changing the "package manager" field in their bit.jsonc), or will be switched on by default if using in a project not created by "bit new".

In "external package manager" mode, the capsules will be installed with npm/yarn and linked after the installation to each other by Bit itself.

Please let me know if I missed anything. :)

[GiladShoham]

@imsnif what about old workspaces?

[davidfirst]

Regarding the IDE experience. I see an issue here.
As discussed, the workspace node_modules will have symlinks to the capsules.
So, if you click on the dependency in the IDE, it'll go to the file in the capsule. My concern is that users will edit this file, thinking it's their source file inside the same project and then, the next time the capsule is built, their changes will be lost.

@ranm8 , this is part of what I'm working on with the legacy-compiler task. Since the dists are not written in the workspace but in the capsule, there must be a link from node_modules to the capsules. Otherwise, it's impossible to compile the source as the import/require statements are invalid.

[GiladShoham]

@davidfirst I talked with @ranm8 about it few days ago. This should be handled by some source maps in the capsule redirect to the source in workspace. But it's still need to be researched