Cache: Cache multiple paths

[hugovk]

It can be useful to cache multiple directories, for example from different tools like pip and pre-commit.

With Travis CI (ignoring their pip: true shortcut):

cache:
  directories:
    - $HOME/.cache/pip
    - $HOME/.cache/pre-commit

The Actions equivalent requires adding a duplicated step for each directory, something like:

      - name: pip cache
        uses: actions/cache@preview
        with:
          path: ~/.cache/pip
          key: ${{ matrix.python-version }}-pip

      - name: pre-commit cache
        uses: actions/cache@preview
        with:
          path: ~/.cache/pre-commit
          key: ${{ matrix.python-version }}-pre-commit

Perhaps also allow multiple directories in a single step, something like this?

      - name: pip cache
        uses: actions/cache@preview
        with:
          path:
            - ~/.cache/pip
            - ~/.cache/pre-commit
          key: ${{ matrix.python-version }}

[chrispat]

We made a scoping call for the v1 to not support multiple paths. This is something we do plan to look at for the future.

[nathany]

Same requirement for Elixir where we commonly cache multiple paths:

paths:
  - _build
  - deps

For now something like this is required:

- uses: actions/cache@preview
  with:
    path: deps
    key: ${{ runner.os }}-mix-${{ hashFiles('**/mix.lock') }}
    restore-keys: |
      ${{ runner.os }}-mix-
- uses: actions/cache@preview
  with:
    path: _build
    key: ${{ runner.os }}-build-${{ hashFiles('**/mix.lock') }}
    restore-keys: |
      ${{ runner.os }}-build-

[mrmckeb]

This will definitely be required for many open source projects. We want to trial this on some Storybook projects, but caching for a monorepo (Yarn) would be needed for that - along with #6.

But that's not to discourage this great first-step.

[Siedlerchr]

This would be very important, for example if you build a web app for android in node, you will add node_modules and also gradle to the cache.

[dbrockman]

Would it be possible to use a glob pattern for path?

The use case that I have in mind is caching the typescript build output in a monorepo.

For example:

path: packages/*/{build,tsconfig.tsbuildinfo}

[smorimoto]

Is anyone already working on this?

[mrmckeb]

It might also be good if this didn't require config for something like Yarn, where you could just say workspaces: true.

[jcornaz]

It might also be good if this didn't require config for something like Yarn

I don't think it is good for this project to have tool-specific behavior. I think it should be agnostic of whatever tools we use.

[smorimoto]

I agree with @jcornaz.

[mrmckeb]

I'm not sure I can agree. That would mean considerable overhead for projects - the bigger they are, the more work someone is going to need to put in to configure their CI.

It would make more sense for the action to do that heavy lifting, as that's exactly what actions exist for.

[smorimoto]

Is this proposal so far from your goal? https://github.com/actions/cache/issues/16#issuecomment-551202039

[smorimoto]

Well, I already say that in #70...

[mrmckeb]

@imbsky, that's a great start - but something like Yarn workspaces has more complex caching requirements.

[smorimoto]

@mrmckeb Yes, I understand that, but do we create options for all ecosystems? That's not realistic.

[jcornaz]

Yes, I understand that, but do we create options for all ecosystems? That's not realistic.

I would go farther than saying "it is not realistic". I would say "it is not desirable".

I think, this this project should strive to be flexible and useful for as many different ecosystems as possible. Introducing yarn-specific behavior, would add maintenance overhead that do not bring benefits to the other ecosystems.

[hugovk]

Certainly not all ecosystems, but for some major ones.

Compare the config for Python's pip with Travis CI:

cache: pip

With GitHub Actions:

      - name: Ubuntu cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'ubuntu')
        with:
          path: ~/.cache/pip
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

      - name: macOS cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'macOS')
        with:
          path: ~/Library/Caches/pip
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

      - name: Windows cache
        uses: actions/cache@v1
        if: startsWith(matrix.os, 'windows')
        with:
          path: ~\AppData\Local\pip\Cache
          key:
            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/setup.py')
            }}
          restore-keys: |
            ${{ matrix.os }}-${{ matrix.python-version }}-

10 characters vs. 1,000 characters!

I know which I'm be more confident using, and prefer suggesting for projects to add :)

Travis also has shortcuts like:

cache: bundler   # Ruby
cache: cocoapods # Objective-C
cache: npm       # npm
cache: yarn      # yarn
cache: ccache    # C or other C/C++ variants 
cache: packages  # R
cache: cargo     # Rust

The flexibility of GHA's caching is good, but it would be even better to have some user-friendly shortcuts. Thanks!

[smorimoto]

Is that something serious? What if the cache directory changes from a specific version in each ecosystem? I don't think it's necessary...

[jcornaz]

10 characters vs. 1,000 characters!

@hugovk, your example is not using the multiple path feature that we are discussing in this issue!

The whole point of this issue, is to simplify such configuration, by letting the user to specify multiple paths at once. Of course, we all want to have a simpler and more concise way to handle multiple paths.

That does not necessarily imply adding tool specific behavior.

And I understand that you are not happy that pip uses different cache folders depending on the OS. But that's a problem you should report to pip, not here. pip could use a path that remains consistent across operating systems (~/.cache/pip for instance). I don't think it is this responsibility of this project to pay for the design issues of pip.

[hugovk]

Sorry, you're right, that was somewhat off-topic, but it is related: the point is that shortcuts can be provided for common setups (multiple paths for some, different OS paths for others). I do know the point of this issue, I opened it :)

Some tool shortcut would include multiple paths; some would include different paths; whatever it needs.

I'm fine with pip using different paths depending on OS (and I have reported an issue to pip to have a way to find out the path on the current OS, and am working on a PR now :)

However, I'd like the GHA interface to be as simple as possible for the users of GHA. I know some maintainers will be put off by having to include and maintain great big chunks of YAML for things which will be common for many projects.

Like my first post in this issue shows 74 vs. 332 characters.

For example, we could close this issue and let the user write their own code to copy all their cache files and directories into a single directory just before the cache upload, and similarly re-instate them right after the cache download. But shortcuts would be helpful.

[mrmckeb]

I agree with @hugovk, supporting the biggest tools out of the box is ideal.

@jcornaz and @imbsky, that being said, if the end result was like GitLab's, that would be OK:

cache:
  paths:
    - node_modules/
    - packages/*/node_modules/
    - .yarn

But Travis has a simpler:

cache:
  yarn: true

[smorimoto]

Honestly, do you change it so often? I don't think the benefit equals the cost of maintaining it forever. I'm not saying that Node.js projects don't need caching (I also use.), but projects that are really large to need caching are usually not simple to specify in templates like that.

[smorimoto]

The Actions team likes to keep things simple. and I completely agree with that. and if you need a Cache action that supports Monorepo, you can fork this repository or create it from scratch. but, it will take a lot of effort. I guess.

[mrmckeb]

Given that a lot of high-profile open source projects are monorepos, and many use Yarn, I still think it would be great to get in.

Examples include:

• Babel
• Jest
• Storybook
• React and Create React App

[chemitaxis]

Ok @mrmckeb and @imbsky I think that the first step is include something like this:

cache:
  paths:
    - node_modules/
    - packages/*/node_modules/
    - .yarn

And in a future, make something simpler like you said:

cache:
  yarn: true

We are moving from CircleCI to Github Actions, and it's very important for us that some points are ready in the earlier release.

Features like this, or for example, don't have a scope in events like "deployments" make to us consider Github Actions a real Continuous Integration flow, or just something "funny" to auto add Labels and other stuff to our continuous development process.

I don't want to be rude, and sorry if I do, but it's very important for a new product see how it's growing and implementing minimal requirements. Remember that for a business make the change to Github Actions is always dangerous for us.

Have a nice day!!

[mrmckeb]

I don't disagree @chemitaxis, the product is new and a lot will need to change. I'm just pointing out that that the goal should be simplicity, and I'd be happy to contribute that code in to help.

[sakulstra]

cache:
  paths:
    - node_modules/
    - packages/*/node_modules/
    - .yarn

Would a pr in that direction be considered? Working with node, you usually have a lot of caching directories (besides the quoted ones we might have some build cache, test cache(jest/cypress/whatever), lint cache, global package cache, ....

While caching them one by one is certainly possible, it's also very much bloated & unreadable. I'd appreciate solution(like the suggested one) where I would only have to type one line per cache folder, not 5-7 :sweat_smile:

I guess one could live without the glob matching, although it would be nice for workspaces & lerna.

[nathany]

paths instead of path would be a really nice addition.

I think anything beyond that should be a separate issue around simplification.

[nathany]

Rather than directly support different languages in the cache action, would it make more sense if the cache action could be used by other actions (#55)?

E.g. actions/[email protected] could cache the deps and _build folders.

Then cache action only needs to be specified directly if caching folders outside of those that the language actions support.

[kiprasmel]

I've created a PR for this - see https://github.com/actions/cache/pull/180

[ktmud]

For those who in need of configuring multiple cache targets for multiple languages or simply want more flexibility, I created an action called Cached Dependencies that allows you to manage all dependencies and caches in one build step.

This action adds two additional config files for you to define reusable commands and avoid copypasta across workflows:

• .github/workflows/caches.js: for cache configs
• .github/workflows/bashlib.sh: for bash command shortcuts

Then you assemble the commands in one simple build step like this:

steps:
- uses: actions/checkout@v2
- uses: ktmud/cached-dependencies@v1
  with:
    parallel: true
    run: |
      cache-restore npm
      npm install
      cache-save npm

      cache-restore pip
      pip install -r requirements.txt
      cache-save pip

      cache-restore cypress
      cd cypress/
      npm install
      cache-save cypress

Obviously, bashlib is not needed if you only have one workflow. You can read more documentation here or see it in action here.

[joshmgross]

:wave: With https://github.com/actions/cache/pull/212, multiple paths are now supported.

This is currently available in the default branch (-uses: actions/cache@master) and we'll be pushing a v2 tag next week, see #323 for details.