Detox: build_universal_framework.sh fails with yarn workspaces

[birkir]

I have a mono repo (lerna) with two react native apps and some shared packages.

• /packages/app1
• /packages/app2
• /packages/shared

When I do lerna bootstrap, the script build_universal_framework.sh fails in the postinstall.

I found the root cause to be https://github.com/wix/Detox/blob/master/detox/scripts/build_universal_framework.sh#L63

mv: cannot move X to Y: Directory not empty

This is because yarn workspaces will run postinstall at the same time on the two separate packages (app1 and app2) so the check will pass that the framework has not been built and goes ahead and builds it (twice) and the one that takes longer will fail.

Possible fix is to force move (with the -f flag), or create some sort of a lockfile to ensure the framework will only be built once.

On my CircleCI environment, I have a step that does npm install -g detox so that the framework will be prebuilt before going into lerna bootstrap.

[LeoNatan]

Not good. The build script should not be run concurrently.

[kristfal]

Also experiencing this when upgrading to 14.0.3.

It used to work on older versions of Detox, so I'm bisecting trying to find the breaking change.

Update: We only have one instance of Detox across all our packages, so the failure may be unrelated to the issue above.

Update 2: I'm able to install 12.11.1, but not 12.11.2 or later.

Also tried npm install -g detox. This fails in the same way as a local yarn or npm install:

npm -g install [email protected]
/usr/local/bin/detox -> /usr/local/lib/node_modules/detox/local-cli/cli.js

> [email protected] install /usr/local/lib/node_modules/detox/node_modules/dtrace-provider
> node-gyp rebuild || node suppress-error.js

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  ACTION binding_gyp_ndtp_target_build_ndtp .
  TOUCH Release/obj.target/ndtp.stamp

> [email protected] postinstall /usr/local/lib/node_modules/detox
> node scripts/postinstall.js

+ xcodebuild -version
+++ dirname /usr/local/lib/node_modules/detox/scripts/build_framework.ios.sh
++ dirname /usr/local/lib/node_modules/detox/scripts
+ detoxRootPath=/usr/local/lib/node_modules/detox
++ node -p 'require('\''/usr/local/lib/node_modules/detox/package.json'\'').version'
+ detoxVersion=14.0.3
++ echo 14.0.3
++ xcodebuild -version
++ shasum
++ awk '{print $1}'
+ sha1=9438597daf71372b4cbcf71962463a77cd7c7c33
+ detoxFrameworkDirPath=/Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33
+ detoxFrameworkPath=/Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33/Detox.framework
+ main
+ '[' -d /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33 ']'
+ '[' '!' -d /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33/Detox.framework ']'
+ echo '/Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33 was found, but could not find Detox.framework inside it. This means that the Detox framework build process was interrupted.
         deleting /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33 and trying to rebuild.'
/Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33 was found, but could not find Detox.framework inside it. This means that the Detox framework build process was interrupted.
         deleting /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33 and trying to rebuild.
+ rm -rf /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33
+ prepareAndBuildFramework
+ '[' -d /usr/local/lib/node_modules/detox/ios ']'
+ detoxSourcePath=/usr/local/lib/node_modules/detox/ios_src
+ extractSources /usr/local/lib/node_modules/detox/ios_src
+ detoxSourcePath=/usr/local/lib/node_modules/detox/ios_src
+ echo 'Extracting Detox sources...'
Extracting Detox sources...
+ mkdir -p /usr/local/lib/node_modules/detox/ios_src
+ tar -xjf /usr/local/lib/node_modules/detox/Detox-ios-src.tbz -C /usr/local/lib/node_modules/detox/ios_src
+ buildFramework /usr/local/lib/node_modules/detox/ios_src
+ detoxSourcePath=/usr/local/lib/node_modules/detox/ios_src
+ echo 'Building Detox.framework from /usr/local/lib/node_modules/detox/ios_src...'
Building Detox.framework from /usr/local/lib/node_modules/detox/ios_src...
+ mkdir -p /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33
+ /usr/local/lib/node_modules/detox/scripts/build_universal_framework.sh /usr/local/lib/node_modules/detox/ios_src/Detox.xcodeproj /Users/kristianfallro/Library/Detox/ios/9438597daf71372b4cbcf71962463a77cd7c7c33
child_process.js:642
    throw err;
    ^

Error: Command failed: /usr/local/lib/node_modules/detox/scripts/build_framework.ios.sh
    at checkExecSyncError (child_process.js:621:11)
    at Object.execFileSync (child_process.js:639:13)
    at Object.<anonymous> (/usr/local/lib/node_modules/detox/scripts/postinstall.js:2:27)
    at Module._compile (internal/modules/cjs/loader.js:722:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:733:10)
    at Module.load (internal/modules/cjs/loader.js:620:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:560:12)
    at Function.Module._load (internal/modules/cjs/loader.js:552:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:775:12)
    at startup (internal/bootstrap/node.js:300:19)

[LeoNatan]

I doubt that. The script hasn't changed in a while, and has never supposed running concurrently.

[LeoNatan]

Seeing as how macOS lacks flock or lockfile for atomic-level file locking, I don't see a way to solve this elegantly. We could write our own binary, but with changes to macOS Catalina that might require notarization even for command-line binaries, we won't be venturing into that adventure anytime soon.

A workaround that I can suggest here is "pre-heating" the Detox cache by installing the Detox version you are using before you run the problematic yarn scenario.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you believe the issue is still relevant, please test on the latest Detox and report back.

Thank you for your contributions!

For more information on bots in this reporsitory, read this discussion.

[PierreCapo]

@birkir can you confirm that a temporary workaround for your CI was to do a
npm install -g detox before doing the npm install ?

[birkir]

@PierreCapo I can confirm.

Here is the part from my circleci.yml

      - run:
          name: Install Detox
          command: |
            HOMEBREW_NO_AUTO_UPDATE=1 brew tap wix/brew
            HOMEBREW_NO_AUTO_UPDATE=1 brew install --HEAD applesimutils
            npm install -g detox-cli
            npm install -g detox
      - run:
          name: Install Dependencies
          command: yarn install --frozen-lockfile

[PierreCapo]

Awesome thanks a lot @birkir I'll try that

[LeoNatan]

Thanks guys.
I think I will close the issue. There isn't really something we can do here that will work in all cases.