Yarn: Yarn upgrade creates duplicate dependency resolution

[xdumaine]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?

I have a dependency tree like this

app
|-- shared-dep@^1.0.0
|-- dep-1
|   |-- shared-dep@^1.1.0
|-- dep-2
     |-- shared-dep@^1.1.0

And when I yarn upgrade shared-dep (say when 1.2.0 is released) it causes duplication in yarn.lock where the resolution for shared-dep doesn't change for the version pulled in by dep-1 and dep-2 and it adds a new (duplicate) resolution for shared-dep so that now it's pulling in 1.1.0 and 1.2.0.

Before upgrade:

"shared-dep@^1.1.0", "[email protected]":
  version: "1.1.0"
  resolved "..."

After upgrade:

"shared-dep@^1.1.0", "[email protected]":
  version: "1.1.0"
  resolved "..."

"shared-dep@^1.2.0":
  version: "1.2.0"
  resolved "..."

If the current behavior is a bug, please provide the steps to reproduce.

1. Clone https://github.com/xdumaine/yarn-issue-3967
2. Run yarn upgrade commander@^2.4.0

3. Observe that the new yarn.lock file has two overlapping entries for commander:

   commander@^2.3.0:
     version "2.3.0"
     resolved "https://registry.yarnpkg.com/commander/-/commander-.3.0.tgz#fd430e889832ec353b9acd1de217c11cb3eef873"
   
   commander@^2.4.0:
     version "2.11.0"
     resolved "https://registry.yarnpkg.com/commander/-/commander-2.11.0.tgz#157152fd1e7a6c8d98a5b715cf376df928004563"
   

What is the expected behavior?

Having a single commander entry in the lockfile:

commander@^2.3.0, commander@^2.4.0:
  version "2.11.0"
  resolved "https://registry.yarnpkg.com/commander/-/commander-2.11.0.tgz#157152fd1e7a6c8d98a5b715cf376df928004563"

Please mention your node.js, yarn and operating system version.

OS X 10.12.5
node v8.9.0
yarn v1.3.2

[BYK]

Hi @xdumaine, thanks for creating a separate issue for this!

Would you mind providing us an isolated, reliable reproduction script so we can test this quickly? Even a very simple package.json file that demonstrates the issue with yarn install would be enough.

Thanks!

[xdumaine]

@BYK This repo has a package.json and yarn.lock which will reproduce it. Just check it out, and run yarn upgrade commander

https://github.com/xdumaine/yarn-issue-3967

[xdumaine]

Note that the results for removing yarn.lock and running yarn install are different from running yarn upgrade when the yarn.lock already exists. Herein lies the bug.

[OliverJAsh]

I can confirm the behaviour described by @xdumaine still occurs on Yarn v1.0.2.

The only workarounds I know of are:

• Manually dedupe by updating lockfile by hand
• Uninstall and then reinstall all dependencies using the shared dependency

[xdumaine]

@BYK can you please update the labels on this so that it will maybe get fixed?

[BYK]

@xdumaine I didn't have time to look into this more yet. Will update the labels once I confirm and know where to look.

[danez]

The current workaround we do is deleting node_modules and yarn.lock and running yarn again.

[voxpelli]

I recently experienced this same behavior with Yarn 1.1.0 in a project and the workaround of deleting node_modules and yarn.lock and running yarn install again solved it for me.

I can't however reproduce the issue by running yarn upgrade commander on https://github.com/xdumaine/yarn-issue-3967 – a simple yarn install even changes the yarn.lock in that repo.

The issue seems to be one of a faulty deduplication – where running a yarn upgrade doesn't properly dedupe subdependencies (and where running it repeatedly doesn't help either).

[xdumaine]

@voxpelli run yarn upgrade commander@^2.11.0 to see the problem.

It's currently still reproducible on yarn 1.1.0 if you do a targeted version upgrade that matches the current package.json range. So yarn upgrade commander in the example repo will not reproduce it, but yarn upgrade commander@^2.4.0 will still reproduce it.

[voxpelli]

I can confirm that yarn upgrade commander@^2.11.0 reproduced the outlined behavior.

Though core issue was about yarn upgrade? Is the expectation that same deduplication should happen when upgrading individual dependencies as when upgrading all dependencies?

Because issue with just yarn upgrade I can't reproduce there.

[xdumaine]

The issue was around upgrading a specific package yarn upgrade shared-dep and still applies. It still requires manual modification of the yarn lockfile every time this happens. I'd really love for the needs-repro-script tag to be removed, because this has a repro script, and can be reproduced every time. @BYK @voxpelli

[voxpelli]

This is probably the same issue that's trying to be fixed in #4636 (which is trying to solve #4476) – right @xdumaine? If so, it looks promising with a fix

[xdumaine]

@voxpelli That issue is marked as resolved, but this still occurs on the latest version of yarn.

[xdumaine]

@wycats This is a major roadblock to us using yarn with ember, and I think it'll cause a lot of problem with projects on 2.16 and module unification because there will be may more dependency trees that look like this.

[BYK]

@xdumaine I've updated your issue to make it clear how to reproduce the issue consistently and what is the expected outcome. Please correct any mistakes you see in them and try to provide exact steps, independent of time to reproduce the issue consistently since I had to read the whole thread to both reproduce the issue and see what to look for in the lock file to understand what is going on.

I agree that this is an important issue to resolve. If this is also a big deal for you, try submitting a PR that fixes it? Even submitting a PR with only a failing test case would help someone else get started on the issue.

[ryank311]

This occurs in our repos as well. The latest versions of yarn (>1.0?) do not correctly resolve transient dependencies, and create duplicates.

For example; our application's package.json pulls in react "^15.6.1" and many of our modules pull in react "^15.6.1 || ^16.0.0". When we delete the yarn.lock and node_modules folder, then run yarn, it generates a yarn lock with both react version "15.6.2" and "16.1.1". In earlier versions of yarn, and our preferred outcome, it should only include version "15.6.2" because that satisfies both dependencies and duplicate packages (especially react) cause all sorts of weird issues and larger bundle sizes when built with webpack.

We also used to use "yarn upgrade" and "yarn upgrade-interactive" but it stopped working for us because we have been seeing the same behavior. Basically, it seems like it no longer re-runs the entire dependency resolution after upgrading a package, and leaves duplicates (dependencies of dependencies) in the node_modules folder, only upgrading the top-most version of that package.

[iHateYourGlasses]

I ran into this as well, after some testing with different versions of yarn (manually deleting yarn.lock and node modules folder and running "yarn" command afterwards) i found out that first version when this bug occurs for me is 0.27.0
For example:
0.26.1

0.27.0

This can be a good starting point to find this bug, right now i'm stuck on 0.26.1 because of it.

[scinos]

You might want to try https://www.npmjs.com/package/yarn-tools to automatically deduplicate the lockfile.

[ryank311]

@scinos we really shouldn't have to use a package to fix the package manager. Yarn has de-dupe commands built in if you use the 'flat' lockfile, but in this scenario, the base behavior is broken.

[scinos]

@ryank311 I fully agree.

However, flat is not the same than this de-duplication. For example

Base package

"shared-dep@^1.1.0"
  version: "1.1.0"
  resolved "..."

"shared-dep@^1.2.0"
  version: "1.2.0"
  resolved "..."

"shared-dep@^2.0.0":
  version: "1.2.0"
  resolved "..."

After de-dupe

shared-dep@^1.1.0, shared-dep@^1.2.0:
  version: "1.2.0"
  resolved "..."

shared-dep@^2.0.0:
  version: "2.0.3"
  resolved "..."

After flat

shared-dep@^1.1.0, shared-dep@^1.2.0, shared-dep@^2.0.0:
  version: "2.0.3"
  resolved "..."

I agree that yarn should do dedupe by default (or at least, have commands for it). Related to #3778

[edmorley]

I'm presuming the below is the same as this issue?

STR:

1. Install yarn 1.4.0
2. Create the following package.json:

{
  "dependencies": {
    "jquery": "3.2.1",
    "jquery.scrollto": "2.1.2"
  }
}

1. And this yarn.lock:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


[email protected]:
  version "2.1.2"
  resolved "https://registry.yarnpkg.com/jquery.scrollto/-/jquery.scrollto-2.1.2.tgz#e7580d9c7ac46ef5bb25319483f6f45713fd7c6c"
  dependencies:
    jquery ">=1.8"

[email protected], jquery@>=1.8:
  version "3.2.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.2.1.tgz#5c4d9de652af6cd0a770154a631bba12b015c787"

1. Run yarn install && yarn upgrade jquery --latest

Expected:

yarn.lock only contains one jquery entry, like so:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


[email protected]:
  version "2.1.2"
  resolved "https://registry.yarnpkg.com/jquery.scrollto/-/jquery.scrollto-2.1.2.tgz#e7580d9c7ac46ef5bb25319483f6f45713fd7c6c"
  dependencies:
    jquery ">=1.8"

[email protected], jquery@>=1.8:
  version "3.3.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.3.1.tgz#958ce29e81c9790f31be7792df5d4d95fc57fbca"

Actual:

Multiple jquery entries in yarn.lock:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


[email protected]:
  version "2.1.2"
  resolved "https://registry.yarnpkg.com/jquery.scrollto/-/jquery.scrollto-2.1.2.tgz#e7580d9c7ac46ef5bb25319483f6f45713fd7c6c"
  dependencies:
    jquery ">=1.8"

[email protected]:
  version "3.3.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.3.1.tgz#958ce29e81c9790f31be7792df5d4d95fc57fbca"

jquery@>=1.8:
  version "3.2.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.2.1.tgz#5c4d9de652af6cd0a770154a631bba12b015c787"

[aikar]

Any update on this instead of having to resort to another tool to repair the lockfile?

The only solution I can see at the moment is to rerun yarn upgrade without any args, and let it reanalyze all versions, but this will update other deps too.

[cordoval]

I tried to use yarn-tools to list the duplicates but instead i got nothing

[email protected]:
  version "3.2.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.2.1.tgz#5c4d9de652af6cd0a770154a631bba12b015c787"

jquery@>=1.2.3, jquery@>=1.7, jquery@x.*:
  version "3.3.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.3.1.tgz#958ce29e81c9790f31be7792df5d4d95fc57fbca"

Basically it installs (my suspicion) 3.3.1 and breaks my system?

[xdumaine]

yarn-tools doesn't work to fix this. It essentially reverts the upgrade.

[cordoval]

I did fix it doing a trick

"resolutions": {
    "jquery": "3.2.1"
  },

[jedwards1211]

I had hoped that yarn add mypackage@^<newversion> would be a workaround, but unfortunately it bungles the lockfile in the same way as yarn upgrade.

[ryankshaw]

this is what we use to get around this issue:

rm -rf yarn.lock node_modules && yes 1 | yarn install --flat --production --ignore-scripts && git checkout package.json && yarn install

yes, it's hacky, and it tries to upgrade everything, but the net result is that we get everything upgraded to the latest versions we allow and everything shares the same copy of each dependency whenever possible.

fyi, I think this issue is really just a side effect of the bigger issue of https://github.com/yarnpkg/yarn/issues/579

[OliverJAsh]

My workaround using yarn-tools:

yarn-tools fix-duplicates yarn.lock > yarn-new.lock && mv yarn-new.lock yarn.lock && yarn --ignore-engines

[JounQin]

Or is there a command to detect duplicate dependencies? When I use resolutions with yarn-tools, it complains about version mismatch.

[Lyfexu]

This looks like intended behavior according to #79. I think there even is a code snippet that explicitly ensures it: code

// don't optimize things that already have a lockfile entry:
// https://github.com/yarnpkg/yarn/issues/79
const collapsablePatterns = patterns.filter(pattern => {
  const remote = this.patterns[pattern]._remote;
  return !this.lockfile.getLocked(pattern) && (!remote || remote.type !== 'workspace');
});
if (collapsablePatterns.length < 2) {
  return;
}

If this is indeed a bug it should be easy to fix. Just remove the lockfile condition from the filter and activate the optimization by default and not only to flat installations (code).

@BYK With special regards to #79, can you please (again) confirm that this is a bug and that it is legitimate to fix it with the proposed changes.

[sudoJoe]

This issue has not been fixed?

[AlexWayfer]

This issue has not been fixed?

Status "Open" of an issue means "not fixed".

[sudoJoe]

I really set myself up there, didn't I?

I was so tired. My apologies. I didn't mean to write a comment at all.

[hubgit]

I can confirm that temporarily adding the dependency to resolutions and running yarn merges
the duplicate entries in yarn.lock. The dependency can then be removed from resolutions.

[edit: or just use yarn-deduplicate]

[Bnaya]

Use yarn-deduplicate

[wKich]

Or npm and throw yarn away.

[jaysonvirissimo]

Can confirm this is still a thing in 1.16.0.

[OliverJAsh]

Reading through this issue to understand the current status, it seems like we're waiting for a reply to https://github.com/yarnpkg/yarn/issues/3967#issuecomment-468962380. /cc @BYK

Upgrading dependencies is a frequent task, and it's a real pain to have to remember to use one of the mentioned workarounds (yarn-tools/yarn-deduplicate/resolutions).

A large number of people have upvoted this issue, so I think it's really important that we get this fixed?

[BYK]

@OliverJAsh thanks for the useful heads up. I'd say this behavior makes sense when running yarn add to keep the already resolved dependencies stable but it doesn't make much sense with yarn upgrade, especially when the intention is to upgrade indirect dependencies.

Maybe we can add a new flag, --recursive to force a behavior similar to resolutions? @rally25rs any thoughts as I remember you working on upgrade some time ago?

[NikitaIT]

i use rush + yarn "rushVersion": "5.13.0", "yarnVersion": "1.10.1",
when i add deps or remove deps:
1) or nothing happens at all
2) either I get something like this with webpack 4.39 in package.json:

[email protected], webpack@^4.0.0:
  version "4.40.1"
  resolved "https://registry.yarnpkg.com/webpack/-/webpack-4.40.1.tgz#26ec01fc9ee7ea5ab5ec1d4ab95cb185ac20c116"
  dependencies:
    ...
[email protected]:
  version "4.40.2"
  resolved "https://registry.yarnpkg.com/webpack/-/webpack-4.40.2.tgz#d21433d250f900bf0facbabe8f50d585b2dc30a7"
  dependencies:
    ...

// webpack@^4.0.0 by deps from vue-cli
// [email protected] its first installed version
// [email protected] its second installed version
// [email protected] its last installed version, haha, what happens where is 4.39.0

I don’t know the problem in yarn or in rush yarn, but in 3 days I could not find a workaround for this problem.

[aaleksandrov]

Still same behaviour in yarn 1.21.1
Was:

aws-sdk@^2.304.0, aws-sdk@^2.560.0, aws-sdk@^2.562.0:
  version "2.562.0"

run yarn upgrade aws-sdk , now I have 2 aws-sdk installations :

aws-sdk@^2.304.0, aws-sdk@^2.562.0:
  version "2.562.0"

aws-sdk@^2.602.0:
  version "2.602.0"

[ocavue]

Yarn 2 has been released. Base on Yarn 2 roadmap and #579 , this issue should be fixed in Yarn 2.

You can install Yarn 2 by following this guide

[Bnaya]

https://www.npmjs.com/package/yarn-deduplicate is still the answer for yarn < 2

[JounQin]

@ocavue yarn@2 and yarn@1 are so different, I don't think everyone will upgrade to yarn@2 immediately.

[AlexWayfer]

Yarn 2 has been released.

Are you sure? I see only RC releases, not stable.

[ocavue]

Hi everyone! After a year of development we've just put Yarn 2 out of beta and into stable.

https://github.com/yarnpkg/yarn/issues/6953#issuecomment-578235326

[AlexWayfer]

Hi everyone! After a year of development we've just put Yarn 2 out of beta and into stable.

#6953 (comment)

OK, big words, but be aware: on the https://yarnpkg.com/package/yarn I see:

RC can be quite stable, but it's not stable release, it's "release candidate".

I have questions to Yarn v2 release actions (updated site for RC), but this issue not about that.