Cache: macos: gtar: permission denied

[aviau]

I am getting the following failures on macos:

/usr/local/bin/gtar --use-compress-program zstd -d -xf /Users/runner/work/_temp/7c5a86f9-4e5e-4c50-b515-6b18d6fcacbe/cache.tzst -P -C /Users/runner/work/perkeep/perkeep
/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/media/heif/dumpheif/dumpheif.go: Cannot open: Permission denied
/usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/media/heif/testdata/park.heic: Cannot open: Permission denied
/usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/media/heif/testdata/rotate.heic: Cannot open: Permission denied
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/errorutil/highlight.go: Cannot open: Permission denied
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/legal/legal.go: Cannot open: Permission denied
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/legal/legal_test.go: Cannot open: Permission denied
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/fault/fault.go: Cannot open: Permission denied
Error: /usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/syncutil/once_test.go: Cannot open: Permission denied
/usr/local/bin/gtar: ../../../go/pkg/mod/[email protected]/syncutil/singleflight: Cannot mkdir: Permission denied

Using cache like so:

    - uses: actions/cache@v2
      with:
        path: ~/go/pkg/mod
        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
        restore-keys: ${{ runner.os }}-go-

Looks similar to https://github.com/actions/cache/issues/133 ?

[mathetake]

I got this too. I rerun a previously successful job, but it also is failing due to this error in a last few hours.

[ncw]

This is also affecting us here: https://github.com/rclone/rclone/runs/1838213713?check_suite_focus=true

The 1000s of errors all take this form

/usr/local/bin/gtar: ../../../go/pkg/mod/github.com/pkg/[email protected]/examples/buffered-write-benchmark: Cannot mkdir: Permission denied

[konradpabjan]

For the moment you can pin to @actions/[email protected] to use an older version that doesn't have this error. We just released a new version yesterday v1.4.0 that switched to using GNU tar on Mac if available: https://github.com/actions/toolkit/pull/701

A few questions that might help narrow this down. Are you using self-hosted runners when getting this error? Any containers?

[gssbzn]

We are facing this error and we don't have anything special, https://github.com/mongodb/mongocli/blob/master/.github/workflows/go-test.yml

[mathetake]

I experienced with exactly the same setting as @.gssbzn has.

[alfredodeza]

Workaround: We experienced the same problem and got around it by pinning to the previous release: actions/[email protected]

Affected tags: Note that the release that introduced the problem is v2.1.4 so if you are using the rolling tag v2 you are implicitly using v2.1.4.

[konradpabjan]

For the moment I've moved the v2 tag back to v2.1.3 since this does seem like a pretty big error. This doesn't follow our normal flow of having a rolling v2 tag that points to the latest tag but we're making an exception for this. Pinning this issue for visibility

[AJMansfield]

Is this related to #404 or #324?

[dscho]

We are facing this error and we don't have anything special, https://github.com/mongodb/mongocli/blob/master/.github/workflows/go-test.yml

@gssbzn this is helpful, I think. Looking at https://github.com/mongodb/mongocli/runs/1861698917?check_suite_focus=true, I see that the cache is restored from a .tzst that was created by BSD tar:

/usr/local/bin/gtar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'

So I wonder whether this has anything to do with the issue: maybe GNU tar has problems with restoring permissions when unpacking .tar files generated using BSD tar (i.e. libarchive)?

FWIW my stake in this is not to make actions/cache work on macOS. I came to this ticket because I was investigating a problem on _Windows_ (concretely, the bug where only paths without slashes would work) and found the actions/[email protected] _fixes_ that problem, so that rollback of v2 cost me a bit of time (even if I now understand why it had to be done, and of course I agree with that decision).

So the sooner this issue can be resolved, the better for me.

FWIW I tried to reproduce this issue using this workflow, but I did not even manage to get the warnings about the extended header keyword 'LIBARCHIVE.creationtime':

name: tar-vs-gtar

on: [push, workflow_dispatch]

env:
  DEVELOPER: 1

jobs:
  test-cache:
    runs-on: macos-latest
    steps:
    - name: diagnose tzstd problems with ~/a/b/c
      shell: bash
      run: |
        set -x
        mkdir -p ~/a/b/c/github.com/go-test/[email protected]/test/v2/
        echo Hello >~/a/b/c/github.com/go-test/[email protected]/test/v2/errors.go
        find ~/a/ -exec chmod a-w {} \;
        ls -laR ~/a
        echo ===
        which tar
        echo ===
        tar --version
        echo ===
        which gtar
        echo ===
        gtar --version
        echo ===
        (cd ~/ && pwd -P)
        echo ===
        (cd ../../.. && pwd -P)
        echo ===
        echo '../../../a/b/c' >manifest.txt
        /usr/bin/tar --format=ustar --fflags --use-compress-program "zstd -T0" -cf cache.tzst -P -C ${{github.workspace}} --files-from manifest.txt
        file cache.tzst
        echo ===
        /usr/local/bin/gtar --posix --use-compress-program "zstd -T0" -cf cache.gtar.tzst -P -C ${{github.workspace}} --files-from manifest.txt
        file cache.gtar.tzst
        echo ===
        /usr/bin/tar --use-compress-program "zstd -d" -tvf cache.tzst
        echo ===
        /usr/local/bin/gtar --use-compress-program "zstd -d" -tvf cache.tzst -P
        echo ===
        find ~/a/ -exec chmod a+w {} \;
        rm -rf ~/a
        /usr/local/bin/gtar -p --use-compress-program "zstd -d" -xf $PWD/cache.tzst -P -C ${{github.workspace}}
        echo ===
        ls -laR ~/a
        echo ===

This is really a conundrum. I suspect that there is something funky going on with permissions, maybe the directory already exists and is write-protected? Or maybe the original permissions are something like 0400 instead of 0555?

[twpayne]

tl;dr: this affects Go projects. You need to do the following:

1. Set the GOFLAGS=-modcacherw environment variable in your builds on GitHub Actions.
2. Avoid re-using any existing cached data by changing your cache key from ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} to ${{ runner.os }}-go1.16-${{ hashFiles('**/go.sum') }} (note the change from -go- to -go1.16-) for example (any change in the cache key is fine).

Background:

I'm also hitting this, also trying to cache Go modules.

This is really a conundrum. I suspect that there is something funky going on with permissions, maybe the directory already exists and is write-protected? Or maybe the original permissions are something like 0400 instead of 0555?

Your suspicions are spot-on. By default pkg/mod exists and is write protected, see https://github.com/golang/go/issues/27161 for background info.

You can override this by setting GOFLAGS=-modcacherw as described in the go mod docs:

The -modcacherw flag instructs the go command to create new directories in the module cache with read-write permissions instead of making them read-only. When this flag is used consistently (typically by setting GOFLAGS=-modcacherw in the environment or by running go env -w GOFLAGS=-modcacherw), the module cache may be deleted with commands like rm -r without changing permissions first. The go clean -modcache command may be used to delete the module cache, whether or not -modcacherw was used.

That said, in my own experiments, setting GOFLAGS=-modcacherw in my workflow config with

jobs:
  test:
    env:
      GOFLAGS: -modcacherw

was not sufficient -- I still got the same error.

I think this is because actions/cache is downloading an existing tar file that was created when pkg/mod had read-only permissions.

I could not find a way to clear the actions/cache cache, so I changed the cache key from ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }} to ${{ runner.os }}-go1.16-${{ hashFiles('**/go.sum') }} (note the change from -go- to -go1.16-), and, with that change, both building from an empty and a populated cache succeeded.

Refs https://github.com/twpayne/chezmoi/pull/1049

[twpayne]

As a followup to this, the version number of actions/cache should be included in the cache key, so that new versions of actions/cache never re-use tarballs generated by old versions of actions/cache.

[dscho]

the version number of actions/cache should be included in the cache key, so that new versions of actions/cache never re-use tarballs generated by old versions of actions/cache.

@twpayne or we could detect the presence of those LIBARCHIVE.* pax entries, and in that case avoid using gtar and instead falling back to using BSD tar to extract the archive. That would be fully backwards-compatible and would not cause re-caching just because of using GNU tar.

[twpayne]

I agree it's possible to detect which version of tar to use to restore the cache. That said, cache invalidation is a famously hard problem, and for correctness it's generally better to invalidate a little too eagerly rather than risk re-using a stale value. Realistically, the version number of actions/cache changes rarely, so invalidating the cache when the version number changes is unlikely to be very costly.

[dscho]

@konradpabjan how about just bumping versionSalt? That would have the same effect as @twpayne suggests: avoiding the use of .tar files that were created previously (read: that were created _not_ by gtar)...

[konradpabjan]

So I was thinking about how to fix this and I don't like any of our options. For context this bug only effects macos users which is less than 10% of our traffic.

Options:

• Bump up versionSalt
  
  
  
  • This would invalidate the cache for ALL users (Ubuntu, Windows etc. ) which could be pretty chaotic as people use cache to share things between jobs and all of a sudden previous cache hits would fail
  
  
• Push out 2.1.4 again and tell effected Mac users to update their cache key
  
  
  
  • After 7 days the previous cache would be invalidated and they could move back to the previous cache key
  
  
  • This would not effect any other users such as Ubuntu, Windows etc.
  
  
  • Pretty tedious and making all Mac users update their cache key is a bit extreme and it could be disruptive
  
  
• Somehow add the tar version to the cache version, so if the tar ever changes it would invalidate the cache
  
  
  
  • Haven't figured out a way of doing this
  
  

All these are pretty meh 😞

What I'm leaning towards is pushing out a minimal version of https://github.com/actions/cache/issues/2 where you could view your cache and manually delete the cache for a specific key. With a basic version out, we could push out 2.1.4 again and tell all Mac users to just delete the cache for the specific key in the UI and things will work without them having to change the key in their YAML.

[twpayne]

I would strongly recommend option 1 (bump up versionSalt). Yes, this invalidates the cache for everyone, but this is a cache, not a persistent data store. Anything that uses a cache should work correctly (albeit more slowly) if the cache is not already populated, and the repopulation of the cache should only happen once per project.

I am strongly against pushing a "new" version 2.1.4. Having multiple versions with the same version number is a recipe for confusion and frustration.

That said, it's your project, and I do very much appreciate your work on it - thank you!

[konradpabjan]

What I meant by "Push out 2.1.4", was to just move v2 back to the commit that 2.1.4 is pointing to (instead of 2.1.3 like it is right now). There would be no release and new commits would be needed, just a tag update.

We've never bumped up the versionSalt before so this would be new and it's somewhat unknown just how disruptive or harmless this would be. Given the amount of actions/cache users, I'm steering more towards being cautious as breaking workflows or at least significantly slowing them down for a bit is never fun.

[twpayne]

Do you have some sort of canary system where you can roll out updates to GitHub Actions users in an incremental manner? This would allow you to update versionSalt or move the v2 tag incrementally, only affecting a few users at once and therefore effecting a gradual transition.

[konradpabjan]

We have canary systems and progressive rollout for everything _except_ for publishing first party actions. It's more of a git thing since standard tags + checkout is used in normal repositories. There is no way to say resolve 10% of downloads for a tag to commit X, while the other 90% are resolved to commit Y. So whenever we release a new version and update a tag like v2 to a specific commit, we're moving everyone at once.

I also looked at the perf implications for option 1 (bumping salt). Given the sheer amount of cache entries that would be invalidated at once (this would be in the TBs), there would be a huge spike in the amount of new cache entries being written and we would effectively DDOS ourselves with the huge increase in traffic. The only safe time to bump up the salt seems to be when releasing a new major version (like v3).

Given the circumstances, working on an option for users to manually delete cache entries seems like the best thing (https://github.com/actions/cache/issues/2). It's highly requested, Mac users wouldn't have to change their YAML files and cache keys. Knock out two birds with one stone. After the new functionality is live, then the v2 tag can be moved to point to v2.1.4 again and any stuck users can be redirected to try out our new feature 😄

[twpayne]

Thank you for the detailed investigation and for the insight. I defer to you here. If I can help, just say how.

[dscho]

If you're concerned about DDoSing, please also consider the impact of asking 10% of your users to perform manual steps, without prior warning (the first indication that they need to do anything would be that their builds break).

I would strongly recommend against doing that.

It is always better to put in a bit more effort on a single developer's side when it can save tens of thousands of users several hours of their life. It's not glorious work, and often it comes with technical debt and ugly workarounds, but it is worth it for your users.

In this instance, I would advise to put in the work to make the salt dependent on the operating system and on the exact tar version in use, and then modify the salt _only_ for the gtar/macOS combination.

Is this slightly ugly? Yes. Is it more helpful than any of the options mentioned here? Also yes. Will it be a bit difficult? Yes: You will have to change the salt constant into a function that takes an argument, and then you will have to extend all the function signatures along the call path so that they take that (optional) argument, right up until the caller that knows whether you use gtar or not, and whether this is on macOS or not.

Sure, you can save yourself a couple of days by not implementing such a solution. The price for that decision will be paid by the users, and I don't think that will happen quietly.

Mind you, I have no beef in this because I worked around this problem a long time ago, by dropping @actions/cache and implementing my own GitHub Action (which is the kind of price I was talking about, the price users pay).

It's just that I have a couple of years experience maintaining Git for Windows, and I would be surprised if my prediction re: user satisfaction were off by a lot.

[dhadka]

I played around with this a bit today and found two workarounds. Both result in passing builds using the rclone repo as an example:

• ❌ Original cache action (to confirm I'm actually reproing this issue) - https://github.com/dhadka/rclone/runs/2307908187
• ✔️ Using --delay-directory-restore argument with gtar: https://github.com/dhadka/rclone/actions/runs/733765022
• ✔️ Running tar with sudo: https://github.com/dhadka/rclone/actions/runs/733779651

While I think running as sudo would solve a number of other permissions-related issues we've seen with the cache, not all environments use passwordless sudo (for example, self-hosted runners). See https://github.com/actions/cache/issues/133#issuecomment-629381394 for more details on this topic. So if we were to go with the sudo option, we'd probably need to make that optional via an input.

[dhadka]

Ugh, sorry for the auto close. Will be creating a branch in actions/cache to pull in the v1.0.7 of the cache module, then test out some of these repos to verify the fix.

[dhadka]

Validated the proposed fix with the mongocli repo:

• Populate cache (@v2.1.3) - https://github.com/dhadka/mongocli/actions/runs/734188204
• Reproduce error (@v2.1.4) - https://github.com/dhadka/mongocli/actions/runs/734198611
• Validate fix (@dhadka/update-1.0.7) - https://github.com/dhadka/mongocli/actions/runs/734211647

and rclone:

• Populate cache (@v2.1.3) - https://github.com/dhadka/rclone/actions/runs/734189339
• Reproduce error (@v2.1.4) - https://github.com/dhadka/rclone/actions/runs/734243868
• Validate fix (@dhadka/update-1.0.7) - https://github.com/dhadka/rclone/actions/runs/734308718

If anyone else wants to test the fix before we publish actions/[email protected], change the uses line in your workflow to uses: actions/cache@dhadka/update-1.0.7.

[dhadka]

Merged and updated tags. @v2 is now on @v2.1.5.