React-native: ☂️✅ Green CI: Re-enabling disabled tests

[hramos]

We have a handful of tests that have been disabled on Circle CI over the years. It's time to get these back up and running so we can merge PRs and cut new releases with confidence.

This is an _umbrella_ task intended to track all the work necessary to re-enable these tests. This issue is being kept up to date so new items will be added and completed ones will be (re)moved. Some of the items will spawn separate issues to track work. If you don't see a name assigned to any of the below items and you would like to help out, please volunteer in the comments and reach out to me to get started!

What needs to be re-enabled?

These tests are configured as aliases in the Circle CI config. I'm happy to help with the Circle CI side of things, but for now you can focus on making sure the command/script (e.g. ./scripts/run-ci-e2e-tests.js --ios) succeeds locally.

| Circle CI Job | Test Name | Test Script | Fix? |
| - | - | - | - |
| test_android | Android e2e | node ./scripts/run-ci-e2e-tests.js --android | |
| test_ios | iOS e2e | node ./scripts/run-ci-e2e-tests.js --ios | |
| test_js | JS e2e | node ./scripts/run-ci-e2e-tests.js --js | https://github.com/facebook/react-native/pull/28392 |
| test_ios | Podspecs | ./scripts/process-podspecs.sh | |

[ericlewis]

• CocoaPods Podspecs tests

@hramos is there more detail on what this should do?

[hramos]

I haven’t looked into it recently. There’s a PR out right now that reworks how CocoaPods is supported in this repo, so it’s an open question whether this test is still necessary or sufficient.

Edit: oh, it’s your PR :) Feel free to remove / provide an updated replacement for testing our CocoaPods support as part of your PR.

[gregoryfm]

@hramos I'd like to help with this ./scripts/run-ci-e2e-tests.js --js

[hramos]

@gregoryfm it looks like @ericlewis is already taking a stab at this, through https://github.com/facebook/react-native/pull/23571.

[ericlewis]

Any help with android e2e would be much appreciated tho!

[ericlewis]

@hramos I think the iOS portion is complete too, it’s e2e test consists of setting up a project via react-native init and confirming that we are indeed using the master branch template & that the snapshot lines up. This should indicate a successful react native init on iOS, and in theory, a successful e2e usage of the react-native experience most have.

[ericlewis]

The js e2e tests consist of us being able to bundle the js successfully, and has been completed as well, as marked. But wanted to note how it works for transparency purposes.

[casperboone]

Working on fixing the android e2e tests! Once I've made some (more) progress, I'll give an update :)

[cpojer]

@casperboone @ericlewis thanks so much for your help on this.

@hramos do you think you could give an update about what we still would like to do? I think there is probably a bunch more integration work for Facebook to keep things green but I'm not entirely up-to-date on what that work may be.

[hramos]

Thanks for all your help getting these tests fixed.

I'm posting an update to clarify what needs to be done before this issue can be closed. The podspec tests and Android end-to-end tests remain disabled, and we could use some help getting them green again.

iOS: Podspec tests

These are failing because the yoga pod cannot be built (expand to see logs):

While building module 'yoga' imported from /var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga/App/main.m:3:
In file included from :1:
In file included from /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Products/Release-iphonesimulator/yoga/yoga.framework/Headers/yoga-umbrella.h:13:
/Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Products/Release-iphonesimulator/yoga/yoga.framework/Headers/CompactValue.h:11:10: fatal error: 'cmath' file not found
#include
^~~
1 error generated.
/var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga/App/main.m:3:9: fatal error: could not build module 'yoga'
@import yoga;
~^~
2 errors generated.

WriteAuxiliaryFile /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Intermediates.noindex/App.build/Release-iphonesimulator/App.build/Objects-normal/i386/App.LinkFileList (in target: App)
    cd /var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga
    write-file /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Intermediates.noindex/App.build/Release-iphonesimulator/App.build/Objects-normal/i386/App.LinkFileList

** BUILD FAILED **


The following build commands failed:
    CompileC /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Intermediates.noindex/App.build/Release-iphonesimulator/App.build/Objects-normal/i386/main.o /var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga/App/main.m normal i386 objective-c com.apple.compilers.llvm.clang.1_0.compiler
    CompileC /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Intermediates.noindex/App.build/Release-iphonesimulator/App.build/Objects-normal/x86_64/main.o /var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga/App/main.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler
(2 failures)

Testing with xcodebuild.
-> yoga (1000.0.0.React)
- WARN | source: Git sources should specify a tag.
- WARN | [iOS] license: Unable to find a license file
- ERROR | [iOS] xcodebuild: Returned an unsuccessful exit code.
- NOTE | xcodebuild: note: Using new build system
- NOTE | [iOS] xcodebuild: note: Planning build
- NOTE | [iOS] xcodebuild: note: Constructing build description
- NOTE | [iOS] xcodebuild: warning: Capabilities for App may not function correctly because its entitlements use a placeholder team ID. To resolve this, select a development team in the build settings editor. (in target 'App')
- NOTE | xcodebuild: /Users/hramos/Library/Developer/Xcode/DerivedData/App-fcnbolkgqgfrrdcvttlgzjkxdyvg/Build/Products/Release-iphonesimulator/yoga/yoga.framework/Headers/CompactValue.h:11:10: fatal error: 'cmath' file not found
- NOTE | xcodebuild: /var/folders/82/w8hcc_tj7yq80482srkc1r2x0bj21j/T/CocoaPods-Lint-20190530-11880-m4ez3b-yoga/App/main.m:3:9: fatal error: could not build module 'yoga'

[!] yoga did not pass validation, due to 1 error.
You can use the --no-clean option to inspect any issue.

/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.6.1/lib/cocoapods/command/lib/lint.rb:92:in block in run' /Library/Ruby/Gems/2.3.0/gems/cocoapods-1.6.1/lib/cocoapods/command/lib/lint.rb:60:ineach'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.6.1/lib/cocoapods/command/lib/lint.rb:60:in run' /Library/Ruby/Gems/2.3.0/gems/claide-1.0.2/lib/claide/command.rb:334:inrun'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.6.1/lib/cocoapods/command.rb:52:in run' /Library/Ruby/Gems/2.3.0/gems/cocoapods-1.6.1/bin/pod:55:in

The failure can be reproduced by running ./scripts/process-podspecs.sh on macOS.

[hramos]

I've updated the root issue with the latest list of disabled tests. I have opened https://github.com/facebook/react-native/pull/28392 to make it easier to re-enable tests and iterate on them on your PRs.

[hramos]

JavaScript end-to-end tests

These run as part of the test_js job, and are disabled by default. To run them, edit .circleci/config.yml and set run_disabled_tests to true under the test_js workflow at the end of the file (pending #28392 getting merged):

workflows:
  version: 2

  tests:
    jobs:
...
      - test_js:
          run_disabled_tests: true
          requires:
            - setup_js

Why it's failing

react-native $ node ./scripts/run-ci-e2e-tests.js --js
...
error Cannot find module './Libraries/polyfills/console.js'

The e2e script creates a react-native package and initializes a new app based on the contents of the template/ directory. As part of this process, the script copies over the root's metro.config.js and rn-get-polyfills.js files. However, when rn-get-polyfills.js is interpreted, there are no files at /private/tmp/react-native-RDAxYTVQ/template/Libraries/polyfills/.

[hramos]

iOS end-to-end tests

These run as part of the test_ios job, and are disabled by default. To run them, edit .circleci/config.yml and set run_disabled_tests to true under the test_ios workflow at the end of the file (pending #28392 getting merged):

workflows:
  version: 2

  tests:
    jobs:
...
      - test_ios:
          run_disabled_tests: true
          requires:
            - setup_ios

Why it's failing

react-native $ node ./scripts/run-ci-e2e-tests.js --ios
...
▸ Building Pods/Flipper [Debug]
▸ Check Dependencies
▸ Building Pods/FlipperKit [Debug]
▸ Check Dependencies
▸ Building Pods/Pods-HelloWorld [Debug]
▸ Check Dependencies
▸ Building Pods/Pods-HelloWorld-HelloWorldTests [Debug]
▸ Check Dependencies
▸ Building HelloWorld/HelloWorld [Debug]
▸ Check Dependencies
▸ Running script 'Start Packager'
▸ Linking HelloWorld
⚠️  ld: directory not found for option '-L-L/Users/distiller/Library/Developer/Xcode/DerivedData/HelloWorld-ddyeadlplgsjokalhojpfxozuuyp/Build/Products/Debug-iphonesimulator/CocoaAsyncSocket'

❌  ld: library not found for -lCocoaAsyncSocket



❌  clang: error: linker command failed with exit code 1 (use -v to see invocation)


Testing failed:
    HelloWorld:
        Linker command failed with exit code 1 (use -v to see invocation)
    Testing cancelled because the build failed.

[hramos]

CocoaPods Podspecs tests

These run as part of the test_ios job, and are disabled by default. They lint the Podspec files in the repository.

To run them, edit .circleci/config.yml and set run_disabled_tests to true under the test_ios workflow at the end of the file (pending #28392 getting merged):

workflows:
  version: 2

  tests:
    jobs:
...
      - test_ios:
          run_disabled_tests: true
          requires:
            - setup_ios

Why it's failing

The script refers to a subspec that no longer exists, _ignore_me_subspec_for_linting_:

react-native $ ./scripts/process-podspecs.sh
+ pod lib lint --subspec=_ignore_me_subspec_for_linting_ --verbose --allow-warnings --fail-fast --private --swift-version=3.0 --sources=https://example.com/react-test.git,https://github.com/CocoaPods/Specs.git

  CDN: trunk Relative path: CocoaPods-version.yml exists! Returning local because checking is only perfomed in repo update
[!] Unable to find a specification named `React/_ignore_me_subspec_for_linting_` in `React (1000.0.0)`.

This was removed in https://github.com/facebook/react-native/commit/2321b3fd7f666ce30f5dad4cd2673ddf22972056#diff-66230b3e029caa37b0fbdc8cbd47f4ab. Re-enabling this test will require adding back the _ignore_me_subspec_for_linting_ subspec to React.podspec or React-Core.podspec. Maybe @alloy knows?

[hramos]

Android end-to-end tests

These run as part of the test_android job, and are disabled by default. To run them, edit .circleci/config.yml and set run_disabled_tests to true under the test_android workflow at the end of the file (pending #28392 getting merged):

workflows:
  version: 2

  tests:
    jobs:
...
      - test_android:
          run_disabled_tests: true
          requires:
            - setup_android

Why it's failing

react-native $ node ./scripts/run-ci-e2e-tests.js --android
...
>>>>> Install end-to-end framework



Command failed, 2 retries left
Command failed, 1 retries left
Command failed, 0 retries left
Failed to install appium

[hramos]

https://github.com/facebook/react-native/pull/28392 fixes the JavaScript e2e tests.

[alloy]

Great stuff @hramos, I’ll dive into these soon too 👍

[timkuilman]

iOS end-to-end tests

These run as part of the test_ios job, and are disabled by default. To run them, edit .circleci/config.yml and set run_disabled_tests to true under the test_ios workflow at the end of the file (pending #28392 getting merged):

workflows:
  version: 2

  tests:
    jobs:
...
      - test_ios:
          run_disabled_tests: true
          requires:
            - setup_ios

Why it's failing

react-native $ node ./scripts/run-ci-e2e-tests.js --ios
...
▸ Building Pods/Flipper [Debug]
▸ Check Dependencies
▸ Building Pods/FlipperKit [Debug]
▸ Check Dependencies
▸ Building Pods/Pods-HelloWorld [Debug]
▸ Check Dependencies
▸ Building Pods/Pods-HelloWorld-HelloWorldTests [Debug]
▸ Check Dependencies
▸ Building HelloWorld/HelloWorld [Debug]
▸ Check Dependencies
▸ Running script 'Start Packager'
▸ Linking HelloWorld
⚠️  ld: directory not found for option '-L-L/Users/distiller/Library/Developer/Xcode/DerivedData/HelloWorld-ddyeadlplgsjokalhojpfxozuuyp/Build/Products/Debug-iphonesimulator/CocoaAsyncSocket'

❌  ld: library not found for -lCocoaAsyncSocket



❌  clang: error: linker command failed with exit code 1 (use -v to see invocation)


Testing failed:
  HelloWorld:
      Linker command failed with exit code 1 (use -v to see invocation)
  Testing cancelled because the build failed.

After upgrading a project with version 0.61.2 to 0.62.0 building the app fails with the same error.

[hramos]

That's right, @timkuilman, these tests are broken on master.

[timkuilman]

I'm sorry I wasn't very clear. After doing the upgrade as explained in the React Native Upgrade tool my app was no longer building. I'm not referring to the test.

In my case, the error was fixed by enforcing the new build system of XCode. I had these lines from ios/appName.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings

    <key>BuildSystemType</key>
    <string>Original</string>

[tinyCoder32]

Hi, I have the same issue here: "ld: library not found for -lCocoaAsyncSocket", " 0.62.2".

[tinyCoder32]

For me it only worked when I added:

$(TOOLCHAIN_DIR)/usr/lib/swift5.0/$(PLATFORM_NAME) to the Library Search Paths.

https://github.com/react-native-community/lottie-react-native/issues/575#issuecomment-542049943