Rescript-compiler: Duplicated package installation/handling

[andywhite37]

@bobzhang asked me to create an issue here for a discussion in the ReasonML #general Discord.

The way that npm installs dependencies often causes problems with BS/Reason compiler when two packages want to install the same dependency but at different, possibly incompatible versions.

For example, if you have a core library mycore, and two packages:

• mypackage1 - depends on mycore v1.0
• mypackage2 - depends on mycore v2.0

And an app myapp which depends on mypackage1 and mypackage2. When you run npm install for myapp it might install mycore v1.0 somewhere in node_modules, and mycore v2.0 somewhere else (either at the root, or nested).

When you try to run the BS/reason build, it seems to just choose one version or the other, and you get warnings like this:

Duplicated package: mycore /home/awhite/dev/andywhite37/myapp/node_modules/mycore (chosen) vs /home/awhite/dev/andywhite37/myapp/node_modules/mypackage2/node_modules/mycore in /home/awhite/dev/andywhite37/myapp/node_modules/mypackage2

Other languages solve this type of problem using:

1. dependency version ranges and constraint solvers to decide on a single version to install
2. package sets - only one version of each library is allowed in a package set
3. exact versions with single dependency installs

The key is that you only have one version of each library installed. I would not rely on semantic versioning to aid with this, as that is basically just an honor code. Also, pure JS libraries probably need to allow duplicate nested installs - it's more the BS/reason code that needs more control over library installs.

[vphantom]

For that matter, I get Duplicated package warnings even if both dependencies are of the exact same version (installed with npm link because they're local in my testing).

[OAGr]

I also get a bunch of duplicated package warnings. These run the same versions of the included libraries.

https://github.com/foretold-app/foretold/tree/master/packages/client

[OAGr]

I think this may be leading to a bunch of bsb-watch errors for me. Whenever it runs, there seems to be about a 90% chance it will lead to an error, causing me to have to rebuild the project, which takes some time.

[bobzhang]

I remember this fixed by @Coobaha here https://github.com/BuckleScript/bucklescript/pull/4045
which version are you running

[OAGr]

I think we fixed the issue actually on our end. We were using a monorepo with Lerna for workspaces. This led to a bunch of separate npm_modules directories with similar packages. We changed to use Yarn Workspaces with hoisting. This change was a bit tricky, but now we're not getting this issue anymore.

You can see our codebase here if interested:
https://github.com/foretold-app/foretold

This was the relevant PR that fixed the issue:
https://github.com/foretold-app/foretold/pull/1181

[Coobaha]

Sounds like it was an actual duplicated modules problem, and warnings were correct :)

[OAGr]

It was, though it wasn't obvious to me what the solution was for a while. I ended up posting to the Discord and got some help. There aren't many good tutorials or similar about ReasonML in monorepos yet, that I know of.

[andywhite37]

As long as the dependencies are managed via npm/yarn, this issue is never going to be well solved, because npm is basically built around the idea of duplicate dependencies with different versions, installed per package by default, which simply doesn't work well in compiled languages. Even if you use peerDependencies for your top-level dependencies, they will also bring in their own potentially duplicated dependencies. Every library in the ecosystem would have to be setup to use peerDependencies to give you full control over which versions you wanted to use for everything, but that would be completely unmanageable in javascript land. Other typed languages have solved this using version constraints/solvers, which allows a project and its dependencies to specify an acceptable range of versions, or package sets, where your ecosystem of dependencies is basically locked to a single version of each dependency which are built and tested together for compatibility. I'd recommend package sets personally, but that doesn't help anyone here. Honestly, I'd recommend using or looking at a language like PureScript with spago, or Haskell with stack, which have solved the problem well. At one point I thought it would be interesting to try to implement a package set concept for ReasonML, but it being backed by npm/node_modules makes this pretty unappealing to try to tackle.

[OAGr]

Yarn 2's constraints system looks pretty neat, but it's not obvious how many orgs will actually be shifting to Yarn 2. I believe bucklescript isn't compatible with plug-and-play, so it would be a challenge to use it soon, but maybe as a goal for 1-3 years from now.
https://yarnpkg.com/features/constraints