Cirrus-ci-docs: Yarn Workspace Caching

[jonstuebe]

Question

I have a monorepo with yarn workspaces and have run into an interesting issue that I'm not sure how to resolve. I'm running into an issue where I'm caching the top level node_modules folder but I can't seem an easy way to cache each package's node_module folder as well.

Is there a way to cache multiple folders as part of one cache? Possibly through a glob or something else? If not, do I simply have to create a ton of caches (one per package)?

Thanks!

[fkorotkov]

Hey @jonstuebe,

It's an interesting use case and "create a ton of caches (one per package)" definitely not a way to go. Unfortunately right now the caching instruction can only cache a single folder since we need to support caching outside of the working directory and make sure the cache entry is portable by storing relative paths inside the cached archive.

I wonder if specifying YARN_CACHE_FOLDER and storing it will solve your issue:

env:
  YARN_CACHE_FOLDER: /tmp/yarn-cache
yarn_cache:
  folder: $YARN_CACHE_FOLDER
  fingerprint_script: find . -name "package.json" -exec sha256sum {} \;
  populate_script: yarn install

If not then we'll need to add an option to store a list/glob of folders with a restriction to folders inside the working folder.

[jonstuebe]

Thanks! I'll give it a try and report back

[jonstuebe]

So it kinda helps but it still requires me to do a yarn install in each task that depends on that cache as the actual node_modules folder's are restored from the cache. And just doing a yarn install is taking somewhere around 90 seconds in each task. What would be amazing is if I could do something like this:

node_modules_cache:
  folder:
    - node_modules
    - packages/**/node_modules
  fingerprint_script: yarn.lock
  populate_script: yarn install

[fkorotkov]

Thanks for checking! Created https://github.com/cirruslabs/cirrus-ci-agent/issues/53 to support glob like syntax in the caching instruction:

node_modules_cache:
  folder: **/node_modules
  fingerprint_script: cat yarn.lock
  populate_script: yarn install

Either I or @edigaryev will take a look this week. Will update this issue once the agent change is available in production.

[jonstuebe]

Out of curiosity why checksum the package.json files over fingerprinting the yarn.lock? Wouldn't that mean you wouldn't cache sometimes when you should?

[fkorotkov]

you are right! fixed it.

[fkorotkov]

@jonstuebe the agent got just a new release of the glob support but it's not yet available on Cirrus. You can try to manually test it:

enc:
  CIRRUS_AGENT_VERSION: 1.14.0
node_modules_cache:
  folder: **/node_modules
  fingerprint_script: cat yarn.lock
  populate_script: yarn install

[jonstuebe]

How would I manually test it if it's not available on cirrus yet?

[fkorotkov]

BTW seems with Yarn 2 aka berry something like this is possible:

node_modules_cache:
  folder: .yarn/cache
  fingerprint_script: cat yarn.lock
install_script: yarn install

[jonstuebe]

ya I'd love to but we don't use Yarn 2

[fkorotkov]

@jonstuebe CIRRUS_AGENT_VERSION will override the current default version which is 1.13.0 at the moment.

[jonstuebe]

Getting this guy:

Cannot expand cache folder glob '/**/node_modules' that points above the current working directory /tmp/cirrus-ci-build/

[fkorotkov]

Do you have folder: /**/node_modules with the prefixed /? In this case the agent thinks it's an absolute path.

[jonstuebe]

Nope, for some reason it always thinks it's an absolute path. This worked though 😉

  env:
    CIRRUS_AGENT_VERSION: 1.14.0
  node_modules_cache:
    folder: "tmp/cirrus-ci-build/**/node_modules"
    fingerprint_script: cat yarn.lock
    populate_script: yarn

[jonstuebe]

To make it stranger, those node_modules folders are not being placed back into the proper folders when globbing. I verified this by running ls at the start of one of the tasks.

[fkorotkov]

I'm working on an integration test in order to enable the new agent version by default. Will make sure it all works. Thanks for checking and sorry it didn't work out right away. Will keep you posted.

[jonstuebe]

No worries! Thanks so much for all the hard work!

[fkorotkov]

@jonstuebe in my integration tests things are getting put back properly. We just released 1.14.1 with fixes. Could you please try it? And also when you ran ls where dit the agent restored caches to?

[jonstuebe]

Will give it a go now. When I ran ls they just weren't where they were supposed to be, so I'm not sure where they are.

[jonstuebe]

Still seeing that ls shows no node_modules in various package directories which makes the tasks fail.

[fkorotkov]

@jonstuebe I believe I fixed it in https://github.com/cirruslabs/cirrus-ci-agent/commit/6157e41686c79b8948472351956a9226fa75e292 which is part of 1.14.3 release. Doing some final tests at the moment and if everything will go smooth will switch the agent version today or on the weekend.

[fkorotkov]

@jonstuebe do you mind to share how you set up the repository to have multiple node_modules? I found a YARN monorepo example and it seems it still creates a single node_modules which seems in parity what is written in YARN's documentation that there will be a single node_modules with symlinks. Do you use lerna or something? I'm trying to test the changes on a more or less real project. 😅

[jonstuebe]

For sure! So from what I understand, yarn workspaces v1 work something like this:

On initial install it will look through each package's package.json and install all necessary dependencies to the root level node_modules. _However_, if there are mismatched versions of dependencies between packages (which happens way more than you'd think), then it won't hoist those dependencies but instead keep those in a local node_modules folder inside of the package so that during the module resolution that happens (nodejs), other packages won't see those and accidentally use the wrong version of a package that you are requiring/importing.

Hopefully that helps and if you have any questions let me know!

[fkorotkov]

I think it should be finally working after https://github.com/cirruslabs/cirrus-ci-agent/pull/65

Here is my first run saving 3 folder: https://cirrus-ci.com/task/5976482709241856?command=Upload%20%27node_modules%27%20cache#L0

And here is the re-run that successfully reused it: https://cirrus-ci.com/task/6688716196413440

@jonstuebe please give it a try!

[fkorotkov]

Closing for now since it's working on the example. @jonstuebe please reopen if the issue still persists.