yarn misuses cache on git dependencies

[SEAPUNK]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
misuses cache to install the git dependency at the wrong commit, seems to happen particularly when specifying git branch

If the current behavior is a bug, please provide the steps to reproduce.
sample repo: https://github.com/SEAPUNK/yarn-test-thing

1. yarn add SEAPUNK/yarn-test-thing
2. edit package.json's dependency from SEAPUNK/yarn-test-thing to SEAPUNK/yarn-test-thing#branch
3. edit the yarn.lock file to change the single dependency from what it is to

yarn-test-thing@seapunk/yarn-test-thing#branch:
  version "0.0.1-something"
  resolved "https://codeload.github.com/seapunk/yarn-test-thing/tar.gz/3ea9fe7984b9005f3cc62b14ee4018013ae0db0f"

i.e. what it will be when you run yarn upgrade

1. yarn install
2. inspect node_modules/yarn-test-thing, and see that the files BRANCH_ONE and BRANCH_TWO are not there

What is the expected behavior?
installs the right version at commit 3ea9fe7984b9005f3cc62b14ee4018013ae0db0f

Please mention your node.js, yarn and operating system version.

node.js 8.7.0
yarn 1.2.1
macos high sierra

---

this is particularly annoying when you have committed a commit to the git dependency branch, ran yarn upgrade, check in the new yarn lockfile to the repo you're working on, only for yarn to not install that "resolved" version in the lockfile when someone else checks the code out and runs yarn install

[SEAPUNK]

i'm lazy so i created what i think the tl;dr of the repro steps are, if they don't work, lmk and i'll give you something that should work better

[jwarby]

Also just got bitten by this with a GitHub dependency - CI build failed after pushing, even though yarn.lock was updated with the new commit hash. Worked around the issue for now by making CI pipeline run yarn cache clean after the other steps in my build.

[SEAPUNK]

got bitten by it again; the problem is that yarn is caching the package by its version in package.json, and not by git commit; so it sees in the lockfile that version is 0.0.1-something, looks for the cache for 0.0.1-something of the package, and uses that

the fix for this is to cache these installed-from-git packages by git commit, so in the yarn cache dir it'd look something like git-yarn-test-thing-3ea9fe7984b9005f3cc62b14ee4018013ae0db0f instead of npm-yarn-test-thing-0.0.1-something

[isaachinman]

The methodology outline by @SEAPUNK seems sound, installing from a git repository directly would indicate you are interested in having the latest codebase irrespective of versions/releases.

[cmawhorter]

this seemed to be fixed (in 1.7 i think) but since upgrading to 1.9.4 it is back to broken and worse.

in the prior problematic state, i used to be able to bump the version in the target dep to force yarn to update it, but now that's not working and i've been removing the yarn.lock file to get around the issue.

i admittedly forgot about yarn cache clean so can't comment on that atm.

Edit: yarn clean cache works

[lidel]

Alternative workaround for github dev deps is to define it as URL:

"devDependencies": {
  "yarn-test-thing": "https://github.com/seapunk/yarn-test-thing/tarball/3ea9fe7984b9005f3cc62b14ee4018013ae0db0f/yarn-test-thing.tar.gz",

It works around github-specific handlers and just does a plain HTTP download with a checksum (works fine for me so far).

[FelipeCoimbra]

Has this issue been resolved in recent yarn versions? I'm experiencing this problem using yarn 1.6.0 and wondering if I can solve this by upgrading.

[bantic]

I've observed this behavior as well, with @rondale-sc, using both yarn 1.15.2 and yarn 1.13.0. In our case it appears that the way the dependency is specified mattered. I.e., specifying it as user/repo#SHA does not update as expected (more details below), but specifying as https://github.com/user/repo.git#SHA does work as expected.

The situation:

• package.json specifies a dependency using user/repo#SHA Github URL form

  "devDependencies": {
    "my-dep": "user/repo#SHA"

• Someone else makes a commit that updates the dep SHA in package.json and the resolved sha and url in yarn.lock, but the version of the dependency doesn't change.
• I do git pull and pick up that commit
• I do yarn install, and it does not update the package on disk to the one specified by the sha.

I was able to fix this in 3 different ways:

1. rm yarn.lock && yarn install
2. rm -rf node_modules && yarn cache clean && yarn install
3. Change the dependency specification to more explicit URL form https://github.com/user/repo.git#SHA before making the commit as described above.

Observed with node 8.15 and macOS Mojave.

Updated to point out that we were specifying deps using the short user/repo#SHA Github URL form, but that it works when using the more explicit URL form https://github.com/user/repo.git#SHA.

[bantic]

Another update to the previous comment, with some more explicit reproduction steps. I've tested a bit and confirmed that this issue is indeed (at least for me) limited only to usage of the user/repo#SHA from of the dependency specifier.

To reproduce:

1. Assume that there are two separate checkouts of a given repo called checkout-A and checkout-B. The repo has a package.json with a devDependency pinned to a specific SHA using the github url form, like so: "devDependencies": { "my-dep": "user/repo#SHA-1" }
2. The user with checkout-A edits package.json locally to change the SHA from SHA-1 to SHA-2. The relevant part of their edited package.json now looks like: "devDependencies": { "my-dep": "user/repo#SHA-2" }
3. The user in checkout-A does yarn. This updates the "my-dep" code on disk to SHA-2, as expected, and also modifies the yarn.lock. The user commits the changed package.json and yarn.lock and pushes the commit
4. User in checkout-B does git pull and gets the commit. They then do yarn and the code on disk for "my-dep" does not update to SHA-2 that way it did for user A.
5. User in checkout-B does yarn cache clean && yarn and again, the "my-dep" on disk does not update to SHA-2, that way it would be expected to
6. User in checkout-B does rm -rf node_modules && yarn cache clean && yarn and now the "my-dep" on disk is updated to SHA-2

I ran through the steps above using yarn 1.15.2, node 8.15.1 and macOS Mojave.
I also ran through the steps above using explicit URLs in the form https://github.com/user/repo.git#SHA and the yarn described in step 4 above did work as expected.

[cellvia]

im hitting this issue, bummer as i was trying to move our whole CI process to use yarn.

the error for me is with tags, not specific commit hash:

error Command failed.
Exit code: 1
Command: git
Arguments: fetch --tags
Directory: c:\yarn-cache\v4\.tmp\4f6877135181ea234579e349f25b0ce3
Output:
From https://gitlab.com/xxx/dashboard
 ! [rejected]        v1.0.x     -> v1.0.x  (would clobber existing tag)
 ! [rejected]        v1.x.x     -> v1.x.x  (would clobber existing tag)
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

[pascalw]

I was bitten by this as well. @bantic's observation seems correct, it doesn't occur when the dependency is specified as https://github.com/user/repo.git#SHA instead of user/repo#SHA (same for specifying a branch instead of a SHA in package.json).