Yarn: `yarn upgrade` breaks dependencies

[chinesedfan]

Do you want to request a feature or report a bug?

Bug.

What is the current behavior?

yarn upgrade updates the version of npmlog, but does not check its dependencies.

If the current behavior is a bug, please provide the steps to reproduce.

Suppose you have a very simple package.json file like following:

{
  "dependencies": {
    "npm": "^3.10.5"
  }
}

Run yarn. You will find the folder structure is like:

node_modules
├─[email protected]
│  ├─[email protected]
│  │  ├─[email protected]

Then yarn upgrade.

node_modules
├─[email protected]
│  ├─[email protected] // it is updated, but it requires "gauge": "~2.7.1"
│  │  ├─[email protected]

What is the expected behavior?

Not only update the version of npmlog, but also inner dependencies like gauge.

Please mention your node.js, yarn and operating system version.

node v6.9.1, OSX 0.11.6

[OmerHerera]

I think all these issues are the same, all of them talking about sub-dependencies not updated:
3115
3082
2394

We need to found a solution.

Thanks

[gossi]

A workaround is to rm -rf node_modules/ and yarn install again. The workaround also may help once... I also did this:

yarn install gauge -D

however this forces you to yarn upgrade gauge EVERYTIME after using yarn... and if you forget it you can't go back to your regular workflow.

[ghost]

Re-install entire node_modules does solve the issue. I wonder why no everyone has this issue. Is this because it only affects certain packages?

[bestander]

Can repro on 0.27.

[arcanis]

(note: please disregard my previous - and now deleted - comment, it was about a different issue)

I can't reproduce this issue (#3202). Running yarn upgrade doesn't upgrade npmlog, so gauge doesn't need to be updated either. Am I missing a step?

My repro script is the following:

rm -rf foo

(
    mkdir -p foo && cd foo
    echo '{"dependencies":{"npm":"^3.10.5"}}' > package.json

    debug() {
        jq .version,.dependencies node_modules/npm/package.json
        jq .version,.dependencies node_modules/npm/node_modules/npmlog/package.json
        jq .version,.dependencies node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json
    }

    echo Normal
    ~/yarn/bin/yarn >& /dev/null
    ~/yarn/bin/yarn list > a
    debug

    echo Upgrade
    ~/yarn/bin/yarn upgrade >& /dev/null
    ~/yarn/bin/yarn list > b
    debug

    rm -rf node_modules

    echo Rm
    ~/yarn/bin/yarn >& /dev/null
    ~/yarn/bin/yarn list > c
    debug
)

[bestander]

@arcanis, this can be reproduced by checking contents of
node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json and node_modules/npm/node_modules/npmlog/package.json.

I think the problem is because those deps are bundled and Yarn does not treat them properly at linking phase.

yarn install
cat node_modules/npm/node_modules/npmlog/package.json
cat node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json
yarn upgrade
cat node_modules/npm/node_modules/npmlog/package.json
cat node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json # (not updated)

[JustinLivi]

I'm not 100% certain that an issue I'm encountering is the same one as described here, but I'm seeing unexpected and breaking behavior when using yarn upgrade with linked dependencies. I have multiple linked dependencies, and running yarn upgrade with them linked mangles the linked modules' node_modules directory and in some cases has overwritten the entire contents of the linked directories.

I have not spent significant time yet attempting to nail down the specifics and extent of the issue, but from my initial tests it appears that yarn add may suffer from the same or similar issue. This issue might also have been reported elsewhere, but the state of yarn issues is a bit overwhelming and this seemed a reasonable candidate.

I'm happy to report this elsewhere if there's a more relevant existing thread or if it seems unrelated. If this seems like the appropriate location, I'm also happy to do further research towards describing the behavior.

[bunkscene]

My temporary solution for this is to run:

npm --prefix .\node_modules\npm\node_modules\npmlog install

after I get this error. I found that the removal of node_modules and yarning again was also only a temporary fix. The update above is less painful.

[IAMtheIAM]

I wish they fix this soon. Its so bad to have to reinstall every time I add a new dependency since it automatically does yarn update when running yarn add somepackage

@bunkscene You can do npm i npmlog and it fixes it. What does prefixing it and referencing node_modules do? It didnt work for me when I tried it the way you suggested.

[BYK]

This may be fixed via #4757. Anyone willing to confirm?

[IAMtheIAM]

Hows that fix coming along?

[rally25rs]

@kaylieEB and I took a look at this.

@arcanis is on to the issue:

I think the problem is because those deps are bundled and Yarn does not treat them properly at linking phase.

In the npm 3.10.10 package, npmlog 4.0.0 and gauge 2.6.0 are bundled with it, as seen here: https://github.com/npm/npm/blob/v3.10.10/node_modules/npmlog/node_modules/gauge/package.json#L126

So on initial install those are just extracted from the bundle .tgz file.

---

On upgrade, those packages are also listed as regular dependencies, so Yarn tries to resolve them. For gauge it finds that ~2.7.1 can be installed, and the hoisting process shifts it's location up the dependency tree (in my case it ends up in node_modules/npm/node_modules/gauge. However the "bundled" 2.6.0 version is still in node_modules/npm/node_modules/npmlog/node_modules/gauge because it's in the extracted npm package .tgz. Unfortunately due to the way module resolution works, node.js will find v2.6.0 before moving up the dep tree to find 2.7.1.

I can think of 2 ways to potentially fix this:

• If a dependency is also a bundleDependency, then don't upgrade it. Just use what is bundled.
• ~If a dependency is also in bundleDependency, then upgrade it but don't allow it to be hoisted.~

---

Actually, thinking about this more, the 2nd option above won't work.

npm specifies:

• dependency: npmlog
• bundleDependency: npmlog

npmlog specifies:

• dependency: gauge

So gauge isn't actually a bundleDependency at all, it just happens to be in the npm .tgz file.
If we don't process npmlog since it is a bundleDependency then Yarn shouldn't try to install gauge at all, so I think that might still be an option here... although it means upgrade won't work the same as NPM, but I would debate that if a package is explicitly bundled with a specific version of a dep, then why should we change it? What if the package author had edited the code for that bundled module? We could overwrite their changes.

Anyone want to poke through the NPM source and figure out how they handle this stuff? I wonder if they actually traverse directories of bundleDeps and just look for package.json files to process?

---

Edit:

I went back to NPM to see what the result would be, and it seems like NPM just doesn't upgrade anything.

~/Projects/yarn-test : grep version node_modules/npm/node_modules/npmlog/package.json
        "type": "version"
    "type": "version"
  "version": "4.0.0"

~/Projects/yarn-test : grep version node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json
  "version": "2.6.0"

~/Projects/yarn-test : npm upgrade

~/Projects/yarn-test : grep version node_modules/npm/node_modules/npmlog/package.json
        "type": "version"
    "type": "version"
  "version": "4.0.0"

~/Projects/yarn-test : grep version node_modules/npm/node_modules/npmlog/node_modules/gauge/package.json
  "version": "2.6.0"

So if NPM doesn't upgrade deps that are also bundled, then I don't think we should either. I propose we implement:

• If a dep or devDep is also a bundleDep, then don't install or upgrade it. Just use what is bundled.
• Be careful not to break #886 (warn if a bundleDep is missing from the package .tgz)

Any volunteers? :)

[JasonKleban]

Sounds like good digging! Although I don't follow it due to lack of my own background on yarn and npm. Is there a workaround, like adding a direct dependency of a specific version of a package (guage or npmlog?)? Or does this only affect certain combinations of node/npm/yarn versions?

My team is getting this error all the time now to the point that I'm completely baffled why there isn't more "us too" :+1:s on this issue by now. I'm concluding that it must be something specific to our setup and hopefully avoidable.

[JasonKleban]

Actually that information did help me. For a reason that is lost to history, we have the npm package itself as a sub-sub- ... devDependency and that version of npm is old and is including an old version of npmlog and and old version of gauge. I'm adding npm@latest as a direct devDependency of the project so that the right bits are installed. I'm hopeful that will solve it. I'm not confident about removing the npm dependency itself.

[iarna]

Bundle dependencies are transitive, so bundling npmlog implicitly bundles gauge. This was one of the changes in npm@3.

In npm@2, gauge would have been bundled simply because the tree would have been deep, so it was effectively bundling transitive dependencies, even though it actually failed to do so if you manually flattened your tree.

You probably should be able to update bundled deps (indeed, npm update --depth=999 does this), replacing them such that the version from the bundle isn't used. We did this by recording when a particular dependency in the tree was coming from a bundle in our lockfile. (And not installing the bundled copy when that's not indicated.)

[lano-vargas]

After running yarn upgrade plenty of dependencies were broken after installing one after another dependency I gave up as each time it would complaint about a different one E.G Error: Cannot find module 'babel-plugin-dynamic-import-node/utils', Error: Cannot find module '@babel/plugin-syntax-nullish-coalescing-operator' etc...

Just running yarn install again fixed all the issue for me 👍