Yarn: Yarn add attempts to resolve packages we didn't ask it to install

[steelbrain]

Do you want to request a _feature_ or report a _bug_?
This is a bug report
What is the current behavior?
The current behavior is, if we do lerna add randomcolor and have a package.json file where we have a few modules that are non-existent in the NPM registry, and are local and to be linked by lerna a few seconds later, It tries to resolve those non-existent modules and fails.

We didn't ask lerna to install those modules so it shouldn't care about them, it should only throw the error when it couldn't find one of the packages we asked it to install
If the current behavior is a bug, please provide the steps to reproduce.
Described above
What is the expected behavior?
Described above
Please mention your node.js, yarn and operating system version.
This is not specific to the Node.js version or OS. I'm using [email protected]

[wycats]

I don't think this is quite right, because ultimately yarn wants to understand the complete dependency graph, and the not-yet-linked packages may affect it.

Is there a good place I can learn about the strategy lerna is using to automate the linking process?

[wycats]

cc @tomdale I think you use lerna?

[steelbrain]

@wycats while I do not quite understand deep workings of Yarn, but let's imagine a dependency to be linked x has 10 more dependencies. The thing in this particular scenario is, none of them are a problem because the user did not ask yarn to install any of them, and in cases of links, the linked packages will have their separate dependencies in a different node_modules anyway :)

So I personally think that if Yarn could just ignore it, that would be wonderful and more in line with the NPM behavior

[tomdale]

@steelbrain I think the problem here is that Yarn, by design, treats the package.json as the source of truth. In package.json, specifying a version range implicitly means "the npm registry is the source of truth" (as opposed to a GitHub URL or file path, for example). Putting the packages in package.json with a version tells Yarn "go find these packages on npm", and it rightfully fails when it can't find them.

As a Lerna user myself, my recommendation would be to just publish the packages to npm. If they're not ready for public consumption, you can always just publish a stub package at version 0.0.1. What's the reason they aren't in npm now?

[steelbrain]

@tomdale Now that I'm thinking about it. Instead of publishing meta packages to npm, if we change the order of execution in lerna that is currently

1. Npm install dependencies
2. Link locals

to

1. Link locals
2. Npm install dependencies

Will it work? Because while the stuff will not be on the registry, but will still be available in node_modules/$name/package.json?

[tomdale]

Let me try to make another analogy that I hope helps.

In React, the model is to construct a tree of virtual DOM from component state. This tree of virtual DOM is then diffed against the previous version, and a sequence of update operations is applied to the _real_ DOM. But in this sense, the DOM is just dumb output—the side effect of computing virtual DOM diffs.

In Yarn, the package.json is the source of truth, like component state in React. As you go through package.jsons, you build up a graph of dependencies, which in this analogy would be the virtual DOM. Only at the very end do you persist that graph of dependencies to disk in the form of the node_modules directory.

In this case, asking Yarn to look at the file system as a secondary source of truth is like asking React to sometimes pull data out of the DOM in case a jQuery plugin had changed it. Not only is it an awkward inversion of the model, it is likely to have consequences that prevent many of the performance and correctness benefits of Yarn.

(Which is not to say that the problem isn't worth solving, just that I think we need some way of expressing the desired semantics in the package.json; I don't think falling back to examining the file system is going to pay dividends long term.)

[tomdale]

@steelbrain I made my above comment before reading yours, sorry about that. Your suggestion to link first, then install dependencies, was my first thought too. But I'm not sure it works because I think Yarn would still check the npm registry. Maybe someone who is more familiar with the implementation can chime in.

[jamiebuilds]

The problem here is actually that you can't install individual packages without updating the dependency in package.json. Lerna currently does this:

npm install external-package-1 external-package-2 external-package-3
ln -s ...
ln -s ...
ln -s ...

In yarn you can only either do:

yarn install

Which installs everything or

yarn add/upgrade package-1 package-2 ...

If we could run installs on a limited set of packages that would fix the issue but at the same time I don't want to add that to yarn

[superherointj]

@thejameskyle What is your concern on adding custom multiple modules install to yarn?

[jamiebuilds]

That's not the concern, the concern is allowing an install which adds dependencies to your node_modules that doesn't update package.json and yarn.lock

[jamiebuilds]

Maybe we can add a yarn add <package...> --no-update-package-implicit-deps-something-flag command

[ccorcos]

As I understand it, the reason Yarn will look in the registry first is so that it can generate a proper lock file, and that makes sense.

First, let me describe a typical situation when you're developing two packages together than have a dependency on each other: Suppose you have a package X that depends on Y, and you want to develop a new version of Y. You might clone Y, yarn link it to your local filesystem, and then yarn link Y inside of X. Then you can make changes to both X and Y. But what happens when you add a new dependency to Y? Those changes don't propagate to the X lockfile. But that's fine because X only depends on the old version of Y and we haven't published the new version of Y yet. So we publish a new version of Y and then update X's dependency on Y and then X's lockfile is update appropriately.

That workflow all makes sense to me and its reasonable that there's this limbo state of mis-matched lockfiles because we're amidst making changes.

Now, let me describe the same situation but you're just getting started with building these package. That is, suppose X and Y don't yet exist on NPM and you're just creating the first version. Therefore, if X depends on [email protected], we cannot generate a proper lockfile yet because [email protected] still needs to be developed. The problem is, when we try to link these projects together, we get this error that Y isn't in the registry so we're stuck. So I think it's OK to let these lockfiles go out of sync and leave it to the develop to make sure they know what they're doing. So that right before they publish the very first [email protected] and [email protected], they can run yarn in both to get the latest dependencies.

Let's also example another scenario -- suppose we have these same packages, X and Y, but we never plan on publishing X or Y. Maybe we're building a project for a client and we want to organize the project into separate modules just for the sake of organization and reusability. In this case, we can't use Yarn without actually publishing these packages -- thats annoying.

In the last, what if X and Y were called app and build-system. Those seem like perfectly reasonable package names if I never plan on publishing them. However, both of those package names exist in the registry. And what's going to happen when link build-system to app is Yarn is going to look up build-system in the registry and create a lockfile in app based on the build-system projects that in the registry -- not the one I have in my local filesystem and that certainly isn't the intention.

If you've made it this far, I hope we're on the same page that perhaps Yarn should allow resolving linked local packages. If not, I'm happy to clarify.

So here's what I think ought to be changed:

• if we run yarn link Y; yarn add Y, then Yarn should skip looking up the project in the registry because we intend it to use the package in our local filesystem.
• if we run yarn add Y; yarn link Y, then we need to regenerate the lockfile using the package from our local filesystem, otherwise the package on the registry may be an entirely different package (e.g. if we're using generic local package names like app or build-system)
• if we run yarn inside a project that has linked packages, then we should re-scan the dependencies of the linked project to regenerate the lockfile, ensuring that the linked package is the same version listed in package.json. that way, if we want to publish new versions of both X and Y, we can bump all the versions appropriately (using lerna perhaps) and then re-run yarn in both projects to generate the proper lockfile before committing and publishing.

I'm keep to Lerna working with Yarn ASAP, so let me know what you think and any tips for where to look if you want me to submit a PR.

[ccorcos]

Just forked yarn to try to do this... I don't even know where to start.

[ccorcos]

@thejameskyle can I get some help here?

[jamiebuilds]

@kittens has said to me privately that he does not want to add this feature. We're going to solve Lerna+Yarn issues by merging Lerna into Yarn.

[ccorcos]

that sounds cool! any ideas when that might happen?

[jamiebuilds]

Unsure, I don't have time to work on it now

[ccorcos]

Ah ok. Bummer. Well try to ping here when this gets moving again. :)
On Fri, Dec 2, 2016 at 23:21 James Kyle [e-mail] wrote:

Unsure, I don't have time to work on it now

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/946#issuecomment-264623025, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABth3zu_vK_P-9bQeHL3PFMRvrIM6wl2ks5rERhhgaJpZM4KVUWf
.

[cpojer]

We are planning to this next half but it's not the most hi-pri thing for as (adoption at FB is really important). We also haven't fully mapped out what that would look like, but yes, we are thinking of effectively "merging" lerna into Yarn. Yarn is really great for using dependencies right now but it doesn't do a good job helping you maintain and publish them. We can do a lot better here. If you have ideas on what that may look like or how to get started with this work, let's talk!

[fivetanley]

@cpojer i'd love to hear what this looks like, we could really use this functionality

[cpojer]

Yeah, I could use this functionality yesterday. Let's regroup early next year after the holidays and figure it out together? :)

[jamiebuilds]

@fivetanley If you look at Cargo's workspaces feature, it's going to mostly look the same, just with a few minor API differences borrowed from Lerna and existing Yarn commands.

[ccorcos]

@thejameskyle any suggestions in the meantime? I have a bunch of projects that are interdependent on each other any my strategy right now is just to yarn link all of them, then find any projects that depend on other local projects using file: syntax and then runnning yarn link <project> for all of those. And then I do a third pass and yarn install every project. Its pretty slow but I get the feeling like it could be an easy fix.

My solution to the lockfiles is just to regenerate all of them whenever I change some dependencies.

[seansfkelley]

For those interested, I wrote a pretty hacky proof-of-concept that implements a subset of Lerna's behavior with Yarn (and also adds some other features I needed on the side).

[bestander]

Workspaces will be a Yarn-native solution to Lerna

[richburdon]

@thejameskyle @kittens Guys -- thanks for working on the monorepo issues. The thing is though, lots of us are really struggling with lerna/yarn due to this issue. I.e., totally dependent on lerna for mono repo, want to use yarn (and have the yarn option for lerna ON), but tragically have to manually edit package.json files every time we add a package to the sub package. I.e., yarn is totally utterly useless in this configuration. It seems to me @thejameskyle 's -no-update-package-implicit-deps-something-flag "hack" would completely unblock us.

lerna monorepo is going to be awesome, but can you help us in the meantime?

Cheers.

[michael-yx-wu]

If people are still looking for a temporary solution, we (Palantir) are currently using yerna until the workspaces stuff ships.

[BYK]

@michael-yx-wu do the 0.28 release or the nightlies not fit the bill for you? If so what are the missing features?

[michael-yx-wu]

Didn't realize that had shipped already. Could you point me to some docs (or the original PR is fine if no docs). I'm having trouble finding the docs online found the blog. is that the only source of docs so far?