Yarn: workspaces - not linking .bin and not installing unhosited package if top level node_modules was cached

[Bnaya]

Do you want to request a feature or report a bug?
bug

Prologue:
CI services, (In my case codeship) are caching the top level node_modules dir on the project
To speedup future npm/yarn installs

What is the current behavior?

• On the first run on the CI, when's there's no node_modules dir, yarn will go and link the needed bins to the sub packages, install unhoisted deps, etc
• On a repetitive run of the CI, yarn install will look at the cached node_modules and will think he is all good and won't perform the bin linking and unhoisted install

What is the expected behavior?
yarn should link and install unhoisted deps for the workspaces

Further thinking
This issue also happens not only when the node_modules was cached, but if for some reason i've delete packages dir and checkedin it again from git and similar scenarios

Immediate workaround
Delete the top level node_module dir/disable CI node_module cache. but not yarn cache!
For my project yarn install is very quick so i can live with that for now

Please mention your node.js, yarn and operating system version.
yarn 0.27.5
node 8.1.2

[BYK]

/cc @bestander

[bestander]

Thanks for reporting.
Yarn has a bug in integrity check for workspaces.
A workaround for you now is rm node_modules/.yarn-integrity before running yarn install.

[arcanis]

After some investigations, a few details:

• The behavior as described in the OP (ie running Yarn with no flags) is not a bug. When yarn starts, it tries to find a yarn-integrity file. If it exists, and if the content of the file match what we should have should we generate a new yarn-integrity file, then we bail out and print "Already up-to-date". The default behavior only include the lockfile entries inside the integrity file, so even if something changes inside the node_modules filesystem, we won't pick it up by default. This is not related to the workspaces, the same thing can be observed with regular installs:

  echo '{"dependencies":{"babel-cli":"*"}}' > package.json
  yarn
  rm -rf node_modules/*
  yarn # Already up-to-date
  

• If you need to bypass this behavior on a one-time basis (for example when running your CI tests), you can disable this logic by using the --skip-integrity-check or --force flags when running yarn install, which will both force Yarn to run the full install whatever the integrity file says.

• If you need to fix this behavior on a day-to-day basis, for example if your filesystem starts removing files at random (in which case I think you'll have more urgent problems :p), you can use the --check-files option, which will add the node_modules recursive directory listing to the integrity file. From now on, Yarn will also check that all of these files exist and, should they not, it will trigger an install as usual. I'm not sure how useful is this flag for your use case, maybe @bestander can give us some insight. I feel like when you reach the point where you need to use it, something is already very wrong.

• That being said, at the present time, the --check-files option doesn't include the workspace files, so that wouldn't help you. That's a bug, it will eventually be fixed.

• Maybe we could also improve the check process by, at the very least, making sure that the node_modules directories exist. It seems innocent enough, shouldn't impact the perfs too much, and could prevent this kind of issues. @BYK wdyt?

[Bnaya]

@aracarie you can get to this state in a very valid workflows. like switching git branches with the same yarn.lock but with new workspace, or on CI that caches top level node_modules (as mentioned)
Right now on my CI i'm deletion node_modules, which is fine for me, and on my machine i'm deleting it form time to time when there's a need.

So if its not a bug, its a design flaw.
With all due respect for yarn, This behavior makes it none deterministic on a reasonable workflow.
checking if node_modules exists still not ensuring that its on the right state.

And thanks for that feature, i'm using it anyhow :)

[arcanis]

like switching git branches with the same yarn.lock but with new workspace

That's actually a very compelling argument for --check-files, good point :)

Don't forget that you can enable this feature by default by adding the following line into your yarnrc:

--check-files true

And until check-files is fixed for the workspaces, you can use --skip-integrity-check instead.

With all due respect for yarn, This behavior makes it none deterministic on a reasonable workflow. Checking if node_modules exists still not ensuring that its on the right state.

Hm I kinda disagree, we're still being deterministic when we detect that an install should happen. The issue is this detection, which isn't that simple (there was no issue before the workspaces, for example - since by reading the integrity file we were effectively checking that the node_modules was there, which is no longer true now that there's multiple node_modules directories).

I think this should eventually be solved on of those two ways:

• Either we check that all the node_modules directories exist (what I was talking in my prev post)
• Or we make --check-files the default when working with workspaces

The second one looks a bit too aggressive. I like the first one.

[Bnaya]

What about some integrity file on each workspace's node_module, that the top level integrity file points to?