Yarn: Installing package from a branch with yarn.lock present installs old version of package

[askielboe]

This is bug report and a follow-up to issue #2280.

Problem:

When installing from a git branch with a correct yarn.lock file present, yarn installs an old version of the package instead of the most recent commit (HEAD), as defined in yarn.lock.

This can lead to inconsistent install states when when using a yarn.lock file.

Steps to reproduce:

The following steps simulate a case where a package has been installed from the master branch of a GitHub repository, and what happens when a user tries to install a new commit to master using yarn.

1. Create a new repository with a single commit of a file version_1.0 and push it to GitHub (in this example askielboe/yarn-bug).

2. Create a package.json file with the following content:

{
  "name": "yarn-bug",
  "version": "1.0.0",
  "dependencies": {
    "yarn-bug": "askielboe/yarn-bug#master"
  }
}

1. Run yarn install at the location of package.json: yarn correctly installs the package with a single file node_modules/yarn-bug/version_1.0.

2. Create a new commit in yarn-bug adding a file version_2.0, and push the commit to GitHub.

3. Run yarn install again: yarn installs from cache without checking for new commits at the origin. This means that the yarn installation of yarn-bug will not include the HEAD commit with the new file version_2.0.

4. Delete node_modules and yarn.lock and run yarn install: With yarn.lock gone, yarn now correctly pulls the latest commit and generates a new yarn.lock file with the HEAD hash.

5. Delete node_modules (but keep yarn.lock) and run yarn install: yarn now installs the initial commit again, instead of the HEAD commit referenced in yarn.lock. Somehow yarn looks at its cache and picks a commit (by random?) to install instead of checking whether it is the HEAD, or the most recent pulled commit.

Notes & workarounds:

• Running yarn upgrade after deleting node_modules will install the correct version of the package.

• Clearing the yarn cache by running yarn cache clean yarn-bug and then yarn install will install the correct HEAD commit without the need to delete yarn.lock.

• Even when using --frozen-lockfile yarn still installs an old commit (ignoring the correct hash in yarn.lock). This can lead to inconsistent state and critical issues in continuous integration, when requiring packages from a git branch.

The bug(s) can be reproduced using the following script: (replace askielboe/yarn-bug with a reference to your own test-repository). Please read through the script before running it, as it creates new directories and clears the yarn cache: https://gist.github.com/askielboe/0c0bc3fdf88193bb17fc4fef5d332ad1

This bug report has been created using yarn 1.2.1 and node 8.7.0 on macOS 10.12.6.

[bestander]

Thanks, @askielboe, this is a great bug report.
The core team will look a this eventually but they are very busy at the moment.
Do you think you could spend some time debugging git-resolver.js and see why is it resolving to the wrong revision?

[askielboe]

I'll take a look if I have time.

Meanwhile I've updated the bug report to reflect the fact that you have to do an install between git commits for the bug to arise. I believe it's due to the fact that yarn then keeps references to several commits in its cache, resulting in inconsistent installations.

I've also included a new script to reproduce the bug in this way.

[askielboe]

I've done a bit of debugging. The yarn codebase is a bit opaque to me unfortunately, and I won't be able to spend more time this week, but here is what I've managed to find so far.

It seems that the issue appears when the package is fetched with a manifest where uid is equal to the package version instead of the commit hash. This happens when fresh: false, and makes BaseFetcher create a folder in yarn cache with path $(yarn cache dir)/npm-yarn-bug-0.0.0, excluding a commit hash.

When calling yarn install, yarn will use the cached version in npm-yarn-bug-0.0.0 regardless of when it was created, ignoring the commit hash in yarn.lock. This happens even if there are caches of the correct commit in the yarn cache (i.e. $(yarn cache dir)/npm-yarn-bug-0.0.0-13dba66e55f4b8b8259577f417d73aef1809c9d8).

If the cache folder npm-yarn-bug-0.0.0 is deleted, yarn installs the correct version until a new npm-yarn-bug-0.0.0 folder is generated.

This is the manifest I get when yarn creates the non-hashed cache folder:

{ name: 'yarn-bug',
    version: '0.0.0',
    _reference:
     PackageReference {
       resolver: [Object],
       lockfile: [Object],
       requests: [Array],
       config: [Object],
       registry: 'npm',
       version: '0.0.0',
       name: 'yarn-bug',
       uid: '0.0.0',
       remote: [Object],
       dependencies: [],
       permissions: {},
       patterns: [Array],
       optional: false,
       level: 0,
       ignore: false,
       incompatible: false,
       fresh: false,
       location: null },
    _remote:
     { resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       type: 'tarball',
       reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       hash: '',
       registry: 'npm' },
    fresh: false }

And this is the manifest I get when yarn created a cache folder including a git hash (which results in the correct version getting installed):

{ name: 'yarn-bug',
    version: '0.0.0',
    _reference:
     PackageReference {
       resolver: [Object],
       lockfile: [Object],
       requests: [Array],
       config: [Object],
       registry: 'npm',
       version: '0.0.0',
       name: 'yarn-bug',
       uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8',
       remote: [Object],
       dependencies: [],
       permissions: {},
       patterns: [Array],
       optional: false,
       level: 0,
       ignore: false,
       incompatible: false,
       fresh: true,
       location: null },
    _remote:
     { resolved: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       type: 'tarball',
       reference: 'https://codeload.github.com/askielboe/yarn-bug/tar.gz/13dba66e55f4b8b8259577f417d73aef1809c9d8',
       registry: 'npm',
       hash: undefined },
    fresh: true }

The main differences are:

• not working -> working
• uid: '0.0.0' -> uid: '13dba66e55f4b8b8259577f417d73aef1809c9d8'
• fresh: false -> fresh: true
• hash: '' -> hash: undefined

[edmorley]

4722 seems related (CC @SEAPUNK)

[SEAPUNK]

it might be the exact same issue, but not 100% sure
i'll keep my issue up just in case

[bestander]

@askielboe, that is a great analysis and a huge help, thanks a lot for digging that deep.
It looks like the problem is in cache folder for git dependencies, maybe the name is using package.json version instead of git hash.

cc @arcanis as he knows more about Yarn/git integration.

More help getting to the bottom of this is always welcome.

[shtse8]

I have the same issue, any updates for fixing this bug?

[raduflp]

the issue is reproducible to this day :(

[xmnlab]

I have the same issue here also.

[cesarizu]

This is still happening with yarn version 1.15.2.

[sergeyzenchenko]

Same issue for me in the latest version (1.21.1). Any updates on this one?

[jtomaszewski]

Happens to me at 1.22.0 as well.

I go around it with yarn cache clean xxx && rm -rf node_modules/xxx && yarn --check-files but I had it appearing over and over again, which is quite annoying I have to admit.

[dword-design]

Are there any plans on fixing this? Otherwise I'll have to get back to NPM since Yarn literally it installs wrong dependencies on production.

[cah-briangantzler]

From what I see, when the package.json refers to a branch, the commit for the current HEAD is stored in the yarn.lock, then when more commits are added, a yarn install always installed the saved commit instead of the new HEAD.

Any updates on fixing this? been living with constantly doing yarn upgrade my-package while in development.