Cache: Cached executables are not executable

[apaszke]

Thanks a lot for the action, it's been a pleasure to use!

We're using the cache to store some precompiled Haskell artifacts, which include executable files. On Linux everything seems to be running smoothly, but from time to time the macOS build cache seems to get corrupted. Here's an example:

1. A new PR triggers a CI run. For some reason the cache didn't work out (not a big deal), so the artifacts get recompiled. The executables get used in the later stages of the run, so they had to be built correctly.
2. An update to PR is pushed, triggering another run. This one no longer rebuilds from scratch, but it almost immediately fails with the following error (~/.stack is part of the cache):

/Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5: /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5: cannot execute binary file

I've been trying to debug this issue, but I'm running out of ideas for what might be going wrong. Is it possible that the macOS runners are using different CPU architectures? Is executable caching supported?

[BrightRan]

Here has the similar issue:
https://github.community/t/caching-fails-when-bootstrapping-a-compiler-but-only-on-macos/140463

[Martinsos]

We are experiencing exactly the same problem!

Job: https://github.com/wasp-lang/wasp/runs/1399690381?check_suite_focus=true

We are doing the same thing on Travis also, and there are no issues.

[Martinsos]

Temporary fix I used is to add the following step right after the actions/cache step:

# TODO: Remove this step once https://github.com/actions/cache/issues/445 is resolved.
- name: Fix MacOS problem with corrupt cached executable
  run: rm -rf ~/.stack/setup-exe-cache
  if: runner.os == 'macOS'

This removes the problematic executable from the cache and stack manages to do without it (I assume it creates it again). It does not seem to increase the build time.

My run where this worked: https://github.com/wasp-lang/wasp/runs/1399709311?check_suite_focus=true .

EDIT: I changed rm -r to rm -rf, so that command will not fail if directory is not present.

[apaszke]

@Martinsos we're doing the same thing in our repo. Fortunately in this case it doesn't cost us much because the setup executables are small, but I'm afraid that it might hint at a larger problem with this action.

[dhadka]

It sounds like this might be related to a few other reported issues of corrupted executables on MacOS. As far as I an tell, the corruption is caused by BSD tar. No idea why, but the problem seems to go away when switching to GNU tar. See https://github.com/actions/cache/issues/460#issuecomment-728366594 for the workaround.

[PatTheMav]

We encountered the same bug with macOS dependencies pre-compiled on CI where a single dylib out of multiples gets corrupted and half the file is filled with zeroes instead of actual content. It worked fine before and suddenly started to break our packaging.

Using gnu-tar has the unfortunate side effect of requiring other Homebrew-installed dependencies like xz to be installed, which get picked up by some libraries we pre-build on CI (and that we cannot have as the libraries are distributed). As gnu-tar will be preinstalled on runners sometime in January, we need to add these workarounds anyway.

FWIW Homebrew fixed it by running /usr/sbin/purge around archiving calls: https://github.com/Homebrew/brew/pull/6931