Stack: nondeterministic build failure on OS X

[expipiplus1]

General summary/comments (optional)

Sometimes building a local package on an OS X runner on GitHub actions with the error message

/path/to/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4: cannot execute binary file

This isn't much to go on, so any advice on how to further debug this would be useful and I'd be happy to try and see.

An example log https://github.com/expipiplus1/vulkan/runs/1174300593?check_suite_focus=true reproduced here:

 stack build $ARGS
  shell: /bin/bash -e {0}
  env:
    ARGS: --system-ghc --flag vulkan:generic-instances --flag VulkanMemoryAllocator:generic-instances
vulkan-utils         > configure (lib)
vulkan-utils         > /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4: /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4: cannot execute binary file
VulkanMemoryAllocator> configure (lib)
VulkanMemoryAllocator> /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4: /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4: cannot execute binary file

--  While building package VulkanMemoryAllocator-0.3.7 using:
      /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4 --builddir=.stack-work/dist/x86_64-osx/Cabal-3.0.1.0 configure --user --package-db=clear --package-db=global --package-db=/Users/runner/.stack/snapshots/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/pkgdb --package-db=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/pkgdb --libdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/lib --bindir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/bin --datadir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/share --libexecdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/libexec --sysconfdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/etc --docdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/VulkanMemoryAllocator-0.3.7 --htmldir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/VulkanMemoryAllocator-0.3.7 --haddockdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/VulkanMemoryAllocator-0.3.7 --dependency=base=base-4.13.0.0 --dependency=bytestring=bytestring-0.10.10.1 --dependency=transformers=transformers-0.5.6.2 --dependency=vector=vector-0.12.1.2-AWRYcz9jfa25Avs2q9Jg9V --dependency=vulkan=vulkan-3.6.7-8F9WnOyvzjTETzpx68rH8t -fgeneric-instances -f-safe-foreign-calls -f-vma-ndebug -f-vma-recording --exact-configuration --ghc-option=-fhide-source-paths
    Process exited with code: ExitFailure 126


--  While building package vulkan-utils-0.1.2 using:
      /Users/runner/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_3.0.1.0_ghc-8.8.4 --builddir=.stack-work/dist/x86_64-osx/Cabal-3.0.1.0 configure --user --package-db=clear --package-db=global --package-db=/Users/runner/.stack/snapshots/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/pkgdb --package-db=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/pkgdb --libdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/lib --bindir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/bin --datadir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/share --libexecdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/libexec --sysconfdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/etc --docdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/vulkan-utils-0.1.2 --htmldir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/vulkan-utils-0.1.2 --haddockdir=/Users/runner/work/vulkan/vulkan/.stack-work/install/x86_64-osx/1e4d8f609a7002ba3c573304b7462c652bf79168f107d517d320b19d44d266eb/8.8.4/doc/vulkan-utils-0.1.2 --dependency=base=base-4.13.0.0 --dependency=bytestring=bytestring-0.10.10.1 --dependency=extra=extra-1.7.7-Fi81nrP88Zzi5P7uq8giy --dependency=file-embed=file-embed-0.0.11.2-75PinoHy92FvJ16artZzB --dependency=filepath=filepath-1.4.2.1 --dependency=template-haskell=template-haskell-2.15.0.0 --dependency=temporary=temporary-1.3-8J4CbYC9Bae4gdkn7lOATi --dependency=typed-process=typed-process-0.2.6.0-6LkRodBg5zu6U68IVO7kbv --dependency=vulkan=vulkan-3.6.7-8F9WnOyvzjTETzpx68rH8t --exact-configuration --ghc-option=-fhide-source-paths
    Process exited with code: ExitFailure 126
Error: Process completed with exit code 1.

Steps to reproduce

It happens with this CI script in https://github.com/expipiplus1/vulkan https://github.com/expipiplus1/vulkan/blob/bd652725c75d16c979e73f9651a7096963579324/.github/workflows/ci.yml#L96-L171

I don't have an OS X machine with stack installed to check.

Stack version

2.3.1

Method of installation

Github action setup-haskell

[ncaq]

I have problem me too.
I suspect that the cache mechanism is the cause.

[expipiplus1]

@ncaq's suggestion seemed to help

[Zialus]

This does indeed seem to be a problem with the caching mechanism given that the error doesn't happen if the cache is disabled. Should we report this here or is this a bug with stack itself?

[expipiplus1]

Either way, it would be good if stack were a bit more resilient to this kind of cache corruption