Berry: [Feature] Does Yarn 2 have any sort of global cache?
Describe the user story
Yarn 1 had a useful feature — it stored global cache, for example in folder /usr/local/share/.cache/yarn, which could be revealed by running yarn cache dir.
Having cache like this allowed to mount it to CI runner so that all jobs on runner will have shared cache, and it will be much faster rather than to cache Yarn's cache per-job.
Yarn 2 does not have such command.
Describe the solution you'd like
Same thing as before. Global cache which will be used as fallback if project's .yarn/cache is empty.
kirill-konshin
All 16 comments
Note that if you're under OSX enabling the global cache isn't very useful due to the use of copy-on-write (all downloaded files are stored in the global cache first, and we clone them into the local cache afterwards; in result all per-project caches from a single computer user effectively contain a single copy of each package).
arcanis
on 15 Feb 2020
But how to determine where will the global cache be located?
Pipelines run on Linux, just in case.
kirill-konshin
on 15 Feb 2020
The default path is system-specific, but you can set it with globalFolder.
arcanis
on 15 Feb 2020
Got the an error with Storybook 6.0.0-alpha.14 and following settings:
$ yarn config set enableGlobalCache true
$ yarn config set globalFolder /usr/local/share/.cache/yarn/yarn2
storybook: info => Loading presets
storybook: WARN Failed to load preset: "/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-preset.js"
storybook: ERR! Error: Cannot find module '@storybook/addons/package.json'
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
storybook: ERR! at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)
storybook: ERR! at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)
storybook: ERR! at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)
storybook: ERR! at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)
storybook: ERR! at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)
storybook: ERR! at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR! at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR! at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR! at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)
storybook: ERR! at Module.require (internal/modules/cjs/loader.js:1040:19)
storybook: ERR! at require (internal/modules/cjs/helpers.js:72:18)
storybook: ERR! at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)
storybook: ERR! at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR! at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR! at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR! Error: Cannot find module '@storybook/addons/package.json'
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! Require stack:
storybook: ERR! - /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js
storybook: ERR! at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
storybook: ERR! at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)
storybook: ERR! at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)
storybook: ERR! at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)
storybook: ERR! at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)
storybook: ERR! at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)
storybook: ERR! at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR! at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR! at Module.load (internal/modules/cjs/loader.js:1000:32)
storybook: ERR! at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)
storybook: ERR! at Module.require (internal/modules/cjs/loader.js:1040:19)
storybook: ERR! at require (internal/modules/cjs/helpers.js:72:18)
storybook: ERR! at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)
storybook: ERR! at Module._compile (internal/modules/cjs/loader.js:1151:30)
storybook: ERR! at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
storybook: ERR! at Module.load (internal/modules/cjs/loader.js:1000:32) {
storybook: ERR! stack: "Error: Cannot find module '@storybook/addons/package.json'\n" +
storybook: ERR! 'Require stack:\n' +
storybook: ERR! '- /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js\n' +
storybook: ERR! 'Require stack:\n' +
storybook: ERR! '- /usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js\n' +
storybook: ERR! ' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)\n' +
storybook: ERR! ' at Function.module_1.Module._resolveFilename (/builds/web-modules/web-modules-core/.pnp.js:44933:54)\n' +
storybook: ERR! ' at resolveFileName (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:29:39)\n' +
storybook: ERR! ' at resolveFrom (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:43:9)\n' +
storybook: ERR! ' at module.exports (/usr/local/share/.cache/yarn/yarn2/cache/resolve-from-npm-5.0.0-15c9db4d33-2.zip/node_modules/resolve-from/index.js:46:47)\n' +
storybook: ERR! ' at Object.<anonymous> (/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/paths.js:9:32)\n' +
storybook: ERR! ' at Module._compile (internal/modules/cjs/loader.js:1151:30)\n' +
storybook: ERR! ' at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)\n' +
storybook: ERR! ' at Module.load (internal/modules/cjs/loader.js:1000:32)\n' +
storybook: ERR! ' at Function.module_1.Module._load (/builds/web-modules/web-modules-core/.pnp.js:44846:14)\n' +
storybook: ERR! ' at Module.require (internal/modules/cjs/loader.js:1040:19)\n' +
storybook: ERR! ' at require (internal/modules/cjs/helpers.js:72:18)\n' +
storybook: ERR! ' at Object.<anonymous> (/builds/web-modules/web-modules-core/.yarn/$$virtual/@storybook-core-virtual-394b4fc012/4/usr/local/share/.cache/yarn/yarn2/cache/@storybook-core-npm-6.0.0-alpha.14-beaed626f9-2.zip/node_modules/@storybook/core/dist/server/manager/manager-webpack.config.js:26:38)\n' +
storybook: ERR! ' at Module._compile (internal/modules/cjs/loader.js:1151:30)\n' +
storybook: ERR! ' at Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)\n' +
storybook: ERR! ' at Module.load (internal/modules/cjs/loader.js:1000:32)',
storybook: ERR! code: 'MODULE_NOT_FOUND',
storybook: ERR! requireStack: [
storybook: ERR! '/usr/local/share/.cache/yarn/yarn2/cache/@storybook-ui-npm-6.0.0-alpha.14-de956e3b8a-2.zip/node_modules/@storybook/ui/noop.js'
storybook: ERR! ]
storybook: ERR! }
It works fine with local cache.
kirill-konshin
on 19 Feb 2020
https://github.com/yarnpkg/berry/issues/965 something similar
kirill-konshin
on 28 Feb 2020
Thanks @kirill-konshin. Makes more sense to leave my comment there
Js-Brecht
on 28 Feb 2020
@arcanis looks like enableGlobalCache works weird for now.
By observing how Yarn 2 works, I see that it does cache dependencies somewhere in the system even when the option enableGlobalCache is not set. Is there a cross-platform way to determine where? I still would like to mount this directory from CI runner, but I'd like to keep enableGlobalCache setting per project...
kirill-konshin
on 29 Feb 2020
enableGlobalCache seems misleading. Global caching is always enabled. Is this correct?
gabemeola
on 22 Apr 2020
Yes and no - without enableGlobalCache, the global cache will just be an indirection and the files will be copied into a per-project cache (depending on the system it can be zero-cost, thanks to copy-on-write).
With enableGlobalCache we directly reference the global cache into the generated PnP map, and don't generate the local cache. It presents some interesting quirks though, hence why it's not the default.
arcanis
on 22 Apr 2020
Hmmm... but this brings up my original question ))) I'd like to keep enableGlobalCache off (in order to keep PnP map to have only local) but I still would like to mount this indirect, system-dependent cache from container to the runner, so that all jobs will share it.
So is there a way to figure out it's location?
kirill-konshin
on 22 Apr 2020
@kirill-konshin I share the same use case with you that I want to be able to save/attach cache to optimize CI/CD resources.
I ended up caching Yarn's local cache folder and everything works just fine. Why don't you do that, is there any complication or unseen issue, or you just didn't think of it?
Following is how I cache Yarn's local cache folder and the results:
jobs:
build:
runs-on: ubuntu-latest
env:
node-version: 12.x
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Setup Node.js ${{ env.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ env.node-version }}
- id: find-yarn-cache-folder
name: Find Yarn's cache folder
run: echo "::set-output name=path::$(yarn config get cacheFolder)"
- name: Cache Yarn's cache folder
uses: actions/cache@v1
with:
path: ${{ steps.find-yarn-cache-folder.outputs.path }}
key: yarn-cache-folder-os-${{ runner.os }}-node-${{ env.node-version }}-${{ hashFiles('yarn.lock') }}
restore-keys: |
yarn-cache-folder-os-${{ runner.os }}-node-${{ env.node-version }}-
yarn-cache-folder-os-${{ runner.os }}-
- name: Install dependencies
run: yarn install
...
Before applying cache:
441 packages were already cached, 1556 had to be fetchedAfter applying cache:
1994 packages were already cached
Updated CI/CD still bundles my application properly and saves ~85% build time.
phuctm97
on 17 May 2020
Btw, this issue seems not resolved yet, why is it closed, guys?
phuctm97
on 17 May 2020
@phuctm97 in my case yarn config get cacheFolder returns me %PROJECT_DIR%/.yarn/cache which is project's folder, not system-wide cache folder.
@arcanis I think this issue has to be reopened, since it will give a lot of benefits for CI like Gitlab. There's still no solution how to determine which system-wide folder Yarn 2 is using. enableGlobalCache is a bit different, I need both:
- local cache (to cache it using Gitlab, per branch)
- global cache (same as in Yarn 1) to speed up when local cache is cold (first checkout of branch).
kirill-konshin
on 20 May 2020
Again, the global cache is in globalFolder.
arcanis
on 20 May 2020
Oh, so it's used even when enableGlobalCache is false? Sorry for confusion.
kirill-konshin
on 20 May 2020
Is enableGlobalCache should disable on Linux with btrfs, but enable with ext4?
qwIvan
on 25 Dec 2020
Related issues
joshmeads
·
4Comments
larixer
·
4Comments
chrisands
·
3Comments
juanpicado
·
4Comments
Santas
·
3Comments
Most helpful comment
@kirill-konshin I share the same use case with you that I want to be able to save/attach cache to optimize CI/CD resources.
I ended up caching Yarn's local cache folder and everything works just fine. Why don't you do that, is there any complication or unseen issue, or you just didn't think of it?
Following is how I cache Yarn's local cache folder and the results:
Before applying cache:
After applying cache:
Updated CI/CD still bundles my application properly and saves ~85% build time.