Stack: Non-deterministic test suite execution failures

[symbiont-joseph-kachmar]

General summary/comments (optional)

stack test seems to non-deterministically fail when executing a test suite for a multi-project repository with parallelism enabled (e.g. stack build -j8 --test) with the following errors:

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

/tmp/stack-7235c340cf08101f/test-ghc-env: openBinaryFile: resource busy (file is locked)

I haven't been able to pin down a reliable reproduction, however if I were to guess I'd say it's got something to do with the testGhcEnvRelFile, possible either colliding with an existing on in this call:

https://github.com/commercialhaskell/stack/blob/06cda686573cb8615f5f78f845a781fdb55a53d3/src/Stack/Build/Execute.hs#L1965

...or bubbling up from Cabal or ghc somehow?

Steps to reproduce

Unable to reliably reproduce.

Expected

stack build -j8 --test should successfully execute all tests in a multi-project repository.

Actual

stack build -j8 --test fails non-deterministically with an error message about busy resources (locked files).

Stack version

$ stack --version
Version 2.1.3, Git revision 0fa51b9925decd937e4a993ad90cb686f88fa282 (7739 commits) x86_64 hpack-0.31.2

Method of installation

• Compiled from source via cabal-install
• Official binary

[ulidtko]

Can confirm, the same happens in our CI with exactly the same stack --version.

It fails when test dependencies are built and installed.

Currently we have to work around and use stack test -j1 which avoids the issue.

[roberth]

Here's another one https://travis-ci.org/github/fpco/inline-c/jobs/734110470

[wraithm]

I'm experiencing the same issue in my company's CI system with Github Actions.