Yarn: Yarn cache will fill your PC when used with local (file:) packages

[hoegge]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Whenever a local package is rebuild, yarn creates a new hash version in the cache, which is good, since in prior versions, the old version was just used irrespective of making a new version of the package.
But, this eventually fills your hard-drive. Today I found out my yarn cache was around 156 GB and it took me 1,5 hour to delete it! About 300-600 copies of each package (4500 cached folder in total). That is not very smart

If the current behavior is a bug, please provide the steps to reproduce.

1. Create a local package
2. Install it in another project using yarn
3. Make a new version of the package
4. Install it in another project using yarn
5. Check your yarn cache - you now have two versions of the package including all its dependencies in your cache.
   Eventually your hard drive will be full, and you'll find out when there is no room left and it takes you hours to delete the cache folder.

If the same local packages is used in, e.g., 10 different places in your solution, then 10 different hash versions will be created in the cache.

What is the expected behavior?
When a new version of a local package is created and installed/updated/used by any project by yarn, the new version / hash should replace the old version.

Please mention your node.js, yarn and operating system version.
Node: 8.11.3
Yarn: 1.7.0
Win 10 Pro 1709

[hoegge]

It really doesn't make sense at all to cache local packages, unless I've overlooked something. You put a copy on the disk and then afterwards you copy from that cache to the disk. Actually yarn should instead support an option do symlink to local packages, so we wouldn't have to copy first and run linklocal afterwards.

[hoegge]

Actually, it seems like yarn makes a new hash of a local package not only when it has changed (which COULD make sense in some scenarios), but it also calculates a new hash and cache it again just when it is included by another application / solution with a package.json. Even when the contents of the folder is the EXACT same - checked with sha1 and md5 it caches a new version with another hash. Does it use the file access date in the sha1 hash also? (which is the only thing I can think of changes)

[line0]

Still present in yarn 1.9.5.

Take note this happens even when the local packages in question have been linked using yarn link, which doesn't make any sense whatsoever and we inevitably amass millions of cached files that are never even used once.

Worse yet, this also ends up slowing down package installation to a crawl even if installing a totally different package because yarn always creates fresh cached copies for all local packages referenced in dependencies/devdependencies. Since every copy includes a full copy of the packages' node_modules folder, things go south even faster.

[hasparus]

This is of course my local module cached gazillion times.
Yarn: 1.9.4

[deyan-a]

Mine is currently 57GB. Was starting to lose my mind where all my storage space was going

[fungilation]

Please PLEASE just stop caching local packages.

[Globegitter]

I am just running into an issue where I want to install a local package that itself is symlinked and the symlinks are fine in the cache but then when they get copied over to the real node_module they are incorrect as they get copied directly from the cache rather then new symlinks created.

[nojvek]

I just realized our Travis CI build was always finding some changed files in yarn cache and re-uploading.

Overtime things were just getting slower and slower downloading and uploading the cache.

Please please please fix it. In the name of the environment at least, the amount of energy wasted because of this bug 🐛 is probably worth a couple of trees.

[hoegge]

Had forgotten about this and suddenly had a full hard drive. Almost half of my drive was yarn's cache of local modules:

And it takes about 30-60 minutes for yarn to clear the cache - yawn!

[Haemp]

Please put some effort into fixing this issue guys - it had me completely stumped.

[isaachinman]

Still an issue - just discovered my yarn cache was close to 100gb.

[JZBo]

Is there any update for the resolution this issue? This is a completely blocking and breaking problem for anyone using local packages, and it unfortunately leaves us unable to leverage all the other superior features of yarn.

[Haemp]

@JZBo @isaachinman I ended up using yarn workspaces for my local modules instead. Basically I'm using a monorepo setup and have a yarn workspace setup that allows me to refer to the packages by their name instead of the file: syntax.

I know not everyone may have that option but it worked for me.

[isaachinman]

@Haemp Hadn't heard of yarn workspaces before. After reading the docs:

Note that the private: true is required! Workspaces are not meant to be published, so we’ve added this safety measure to make sure that nothing can accidentally expose them.

So it appears this is just for private/local testing?

[Haemp]

@isaachinman Yea that seems about right, it's still pretty new.

[isaachinman]

I don't think it helps me much then. My use case is over at next-i18next where we have an example subdir that consumes the main package.

[mrseanryan]

workaround - use yarn cache clean

example from package.json:

  "devDependencies": {
    "basic-parser": "file:../parsers/basic-parser",
   }

where ../parsers/basic-parser points to another folder that contains another package.json

Then regularly (after each yarn) run yarn cache clean for each local package:

yarn cache clean "basic-parser"

This seems to free up the disk space.

[rohit-gohri]

@mrseanryan Can add that to the "postinstall" script to run it automatically

[mrseanryan]

@rohit-smpx yes - that works too

in the consuming package.json:

  "scripts": {
    "postinstall": "yarn cache clean basic-parser && yarn cache clean my-other-package"
  },

cleaner than a bash script :)

[JZBo]

workaround - use yarn cache clean
.....

Then regularly (after each yarn) run yarn cache clean for each local package:

yarn cache clean "basic-parser"

I'm sorry, but as far as my understanding goes this is by no means a workaround, it's just a way to clean up the yarn cache.

In my opinion, this is not a workaround because it does not address the issue. The local package is still cached, and thus it's both copied to the cache and then written in the package directory. And then you have to spend time to clean it. There's a huge amount of waste.

The point of this issue is that local packages should not be cached at all by yarn.

[mrseanryan]

Alright - no ones suggesting the issue should not be fixed.

I shared the above as it mitigates some of the effects (disk filling) and thought it might be useful, if only as mitigation.

[jackflyer]

It seems that yarn creates different cache of local packages, whenever package.json updated.

[bluelovers]

pls allow set limit size of total cache

[stefan-toubia]

Hey everyone, 448GB checking in here... 🤯
How's that fix coming along?

[velipso]

checking in with 96GB rmed last year, and 70GB rmed this year

"fixed" via:

cd ~/Library/Caches/Yarn
rm -rf *