Nimble: Tests missing during execution

Created on 20 Dec 2016  ·  11Comments  ·  Source: Quick/Nimble
  • [x] I have read CONTRIBUTING and have done my best to follow them.

We use Atlassian Bamboo and have noticed that each of our builds has a different number of executed tests. During the last 6 builds, we have gotten results for 241, 265, 262, 271, 269, and 267 tests. There should have been the same number of tests for each. The last two builds (269 and 267) had identical source code - I just kicked off a new build without adding any new commits. None of these include all of the tests - there should be 306. I have only been able to consistently reproduce this problem on our build machine, though when developing, I run tests via Xcode rather than fastlane. I have seen the issue in Xcode on my development machine as well, but not nearly as often, which is why I think it is a Nimble issue rather than fastlane.

What did you do?

Ran unit tests using fastlane.

What did you expect to happen?

All tests execute during every build.

What actually happened instead?

Only some tests execute during every build.

Environment

List the software versions you're using:

  • Quick: 0.10.0
  • Nimble: 5.1.1
  • Xcode Version: 8.0 (8A218a)
  • xcode-select: 2343
  • Swift Version: Xcode Default
  • Carthage: 0.17.2 (Use carthage version in Terminal)
  • iOS 9.3, iPad 2 simulator
  • fastlane

Project that demonstrates the issue

Because this problem is intermittent and involves our build stack, I did not make a project to demonstrate it.

Test script:
bundle exec fastlane test --env development

Fastfile:

fastlane_version "1.100.0"

default_platform :ios

platform :ios do
  desc "Runs unit tests"
  lane :test do
    scan(scheme: "UnitTests", custom_report_file_name: "junit_unittests.xml", skip_build: true)     #uses defaults from GymFile
  end
end

Gymfile:

scheme "MyScheme"

output_directory "./"
silent true
clean true
output_name "MyApp.ipa" 
use_legacy_build_api false

Scanfile:

scheme "UnitTests"
output_directory "build/reports"
custom_report_file_name "report"
clean true
skip_slack true
configuration "Release"
code_coverage true
device "iPad 2"

Bamboo build log:

simple  19-Dec-2016 17:07:15    Starting task 'test app' of type 'com.atlassian.bamboo.plugins.scripttask:task.builder.script'
command 19-Dec-2016 17:07:15    Beginning to execute external process for build 'My Build - Default Job #6 (MYBUILD-JOB1-6)'\n ... running command line: \n/bin/sh /Users/me/bamboo-agent-home/temp/MYBUILD-JOB1-6-ScriptBuildTask-239867205140649615.sh\n ... in: /Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1\n ... using extra environment variables: \nbamboo_planRepository_1_branch=mybranch\nbamboo_ServerList=\nbamboo_repository_revision_number=6b107cb05c518ec8af1a850b292508356ac4bebb\nbamboo_resultsUrl=https://bamboo.mycompany.com/browse/MYBUILD-JOB1-6\nbamboo_repository_66683133_previous_revision_number=54e7e131baf9ea7fc0bc0a20f55d5854b0bae5d2\nbamboo_capability_system_builder_xcode_iOS_9_3=iphoneos9.3\nbamboo_repository_66683133_git_branch=mybranch\nbamboo_capability_system_builder_xcode_OS_X_10_11=macosx10.11\nbamboo_capability_system_jdk_JDK_1_8_0_77=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/jre\nbamboo_planRepository_1_name=myapp-ios\nbamboo_build_working_directory=/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1\nbamboo_buildKey=MYBUILD-JOB1\nbamboo_ReleaseName=None\nbamboo_DeployConfiguration=\nbamboo_shortPlanName=MyBranch\nbamboo_repository_66683133_branch_name=mybranch\nbamboo_planRepository_name=myapp-ios\nbamboo_buildNumber=6\nbamboo_DeployAppPoolName=\nbamboo_shortJobName=Default Job\nbamboo_buildResultsUrl=https://bamboo.mycompany.com/browse/MYBUILD-JOB1-6\nbamboo_repository_66683133_name=myapp-ios\nbamboo_planRepository_repositoryUrl=ssh://[email protected]:7999/at/myapp-ios.git\nbamboo_agentId=56131585\nbamboo_planName=MyTeam - myapp-ios - MyBranch\nbamboo_shortPlanKey=MYBUILD109\nbamboo_build_commandline_com_atlassian_bamboo_plugins_scripttask_task_builder_script_8=/bin/sh /Users/me/bamboo-agent-home/temp/MYBUILD-JOB1-6-ScriptBuildTask-3863247250435492791.sh\nbamboo_shortJobKey=JOB1\nbamboo_planRepository_revision=6b107cb05c518ec8af1a850b292508356ac4bebb\nbamboo_repository_previous_revision_number=54e7e131baf9ea7fc0bc0a20f55d5854b0bae5d2\nbamboo_buildTimeStamp=2016-12-19T16:31:03.959-05:00\nbamboo_BUILD_CHANGE_LOG_PATH=CHANGELOG.md\nbamboo_planRepository_previousRevision=54e7e131baf9ea7fc0bc0a20f55d5854b0bae5d2\nbamboo_build_commandline_com_atlassian_bamboo_plugins_scripttask_task_builder_script_3=/bin/sh /Users/me/bamboo-agent-home/temp/MYBUILD-JOB1-6-ScriptBuildTask-2351542363174470548.sh\nbamboo_buildResultKey=MYBUILD-JOB1-6\nbamboo_repository_git_branch=mybranch\nbamboo_repository_branch_name=mybranch\nbamboo_buildPlanName=MyTeam - myapp-ios - MyBranch - Default Job\nbamboo_planRepository_1_revision=6b107cb05c518ec8af1a850b292508356ac4bebb\nbamboo_repository_name=myapp-ios\nbamboo_BuildConfiguration=Debug\nbamboo_DeployServer=\nbamboo_RunAutomation=no\nbamboo_capability_system_builder_xcode_Simulator___iOS_10_0=iphonesimulator10.0\nbamboo_planRepository_branch=mybranch\nbamboo_capability_system_builder_xcode_iOS_10_0=iphoneos10.0\nbamboo_agentWorkingDirectory=/Users/me/bamboo-agent-home/xml-data/build-dir\nbamboo_capability_system_git_executable=/usr/bin/git\nbamboo_planRepository_1_previousRevision=54e7e131baf9ea7fc0bc0a20f55d5854b0bae5d2\nbamboo_repository_git_username=\nbamboo_repository_66683133_git_username=\nbamboo_planRepository_1_type=stash-rep\nbamboo_planRepository_branchName=mybranch\nbamboo_TestConfiguration=Debug\nbamboo_capability_system_jdk_JDK=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/jre\nbamboo_capability_system_builder_xcode_Simulator___iOS_9_3=iphonesimulator9.3\nbamboo_IISAppName=None\nbamboo_planRepository_type=stash-rep\nbamboo_planRepository_1_username=\nbamboo_repository_66683133_git_repositoryUrl=ssh://[email protected]:7999/at/myapp-ios.git\nbamboo_repository_git_repositoryUrl=ssh://[email protected]:7999/at/myapp-ios.git\nbamboo_capability_system_builder_node_Node_js=/usr/local/Cellar/node/6.3.1/bin/node\nbamboo_working_directory=/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1\nbamboo_planKey=MYBUILD\nbamboo_planRepository_1_repositoryUrl=ssh://[email protected]:7999/at/myapp-ios.git\nbamboo_planRepository_username=\nbamboo_repository_66683133_revision_number=6b107cb05c518ec8af1a850b292508356ac4bebb\nbamboo_DeployAppName=\nbamboo_JUnit_TestsPath=build/reports/*.xml\nbamboo_capability_system_jdk_JDK_1_8=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home/jre\nbamboo_planRepository_1_branchName=mybranch\n
build   19-Dec-2016 17:07:16    +---------------------------+---------+-------------------+
build   19-Dec-2016 17:07:16    |                      Used plugins                       |
build   19-Dec-2016 17:07:16    +---------------------------+---------+-------------------+
build   19-Dec-2016 17:07:16    | Plugin                    | Version | Action            |
build   19-Dec-2016 17:07:16    +---------------------------+---------+-------------------+
build   19-Dec-2016 17:07:16    | fastlane-plugin-changelog | 0.6.0   | emojify_changelog |
build   19-Dec-2016 17:07:16    |                           |         | read_changelog    |
build   19-Dec-2016 17:07:16    |                           |         | stamp_changelog   |
build   19-Dec-2016 17:07:16    |                           |         | update_changelog  |
build   19-Dec-2016 17:07:16    +---------------------------+---------+-------------------+
build   19-Dec-2016 17:07:16    
build   19-Dec-2016 17:07:16    [17:07:16]: -------------------------------------------------
build   19-Dec-2016 17:07:16    [17:07:16]: --- Step: Verifying required fastlane version ---
build   19-Dec-2016 17:07:16    [17:07:16]: -------------------------------------------------
build   19-Dec-2016 17:07:16    [17:07:16]: Your fastlane version 2.1.1 matches the minimum requirement of 1.100.0  ?
build   19-Dec-2016 17:07:16    [17:07:16]: ------------------------------
build   19-Dec-2016 17:07:16    [17:07:16]: --- Step: default_platform ---
build   19-Dec-2016 17:07:16    [17:07:16]: ------------------------------
build   19-Dec-2016 17:07:16    [17:07:16]: Loading from './fastlane/.env.development'
build   19-Dec-2016 17:07:16    [17:07:16]: Driving the lane 'ios test' ?
build   19-Dec-2016 17:07:16    [17:07:16]: ------------------
build   19-Dec-2016 17:07:16    [17:07:16]: --- Step: scan ---
build   19-Dec-2016 17:07:16    [17:07:16]: ------------------
build   19-Dec-2016 17:07:16    [17:07:16]: Successfully loaded '/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/fastlane/Scanfile' ?
build   19-Dec-2016 17:07:16    
build   19-Dec-2016 17:07:16    +----------------------+---------------------+
build   19-Dec-2016 17:07:16    | Detected Values from './fastlane/Scanfile' |
build   19-Dec-2016 17:07:16    +----------------------+---------------------+
build   19-Dec-2016 17:07:16    | output_directory     | build/reports       |
build   19-Dec-2016 17:07:16    | clean                | true                |
build   19-Dec-2016 17:07:16    | skip_slack           | true                |
build   19-Dec-2016 17:07:16    | configuration        | Release             |
build   19-Dec-2016 17:07:16    | code_coverage        | true                |
build   19-Dec-2016 17:07:16    | device               | iPad 2              |
build   19-Dec-2016 17:07:16    +----------------------+---------------------+
build   19-Dec-2016 17:07:16    
build   19-Dec-2016 17:07:16    [17:07:16]: $ xcodebuild -list -project ./myapp.xcodeproj -configuration Release
build   19-Dec-2016 17:07:16    [17:07:16]: $ xcodebuild clean -showBuildSettings -scheme UnitTests -project ./myapp.xcodeproj -configuration Release
build   19-Dec-2016 17:07:17    [17:07:17]: Successfully loaded '/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/fastlane/Scanfile' ?
build   19-Dec-2016 17:07:17    [17:07:17]: No values defined in './fastlane/Scanfile'
build   19-Dec-2016 17:07:17    [17:07:17]: $ xcodebuild -list -project ./myapp.xcodeproj -configuration Release
build   19-Dec-2016 17:07:18    [17:07:18]: $ xcodebuild clean -showBuildSettings -scheme UnitTests -project ./myapp.xcodeproj -configuration Release
build   19-Dec-2016 17:07:19    
build   19-Dec-2016 17:07:19    +-------------------------+------------------------------------------------------------------------+
build   19-Dec-2016 17:07:19    |                                      Summary for scan 2.1.1                                      |
build   19-Dec-2016 17:07:19    +-------------------------+------------------------------------------------------------------------+
build   19-Dec-2016 17:07:19    | scheme                  | UnitTests                                                              |
build   19-Dec-2016 17:07:19    | custom_report_file_name | junit_unittests.xml                                                    |
build   19-Dec-2016 17:07:19    | skip_build              | true                                                                   |
build   19-Dec-2016 17:07:19    | output_directory        | build/reports                                                          |
build   19-Dec-2016 17:07:19    | clean                   | true                                                                   |
build   19-Dec-2016 17:07:19    | skip_slack              | true                                                                   |
build   19-Dec-2016 17:07:19    | configuration           | Release                                                                |
build   19-Dec-2016 17:07:19    | code_coverage           | true                                                                   |
build   19-Dec-2016 17:07:19    | device                  | iPad 2                                                                 |
build   19-Dec-2016 17:07:19    | project                 | ./myapp.xcodeproj                                                   |
build   19-Dec-2016 17:07:19    | derived_data_path       | /Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/De... |
build   19-Dec-2016 17:07:19    | address_sanitizer       | false                                                                  |
build   19-Dec-2016 17:07:19    | output_types            | html,junit                                                             |
build   19-Dec-2016 17:07:19    | buildlog_path           | ~/Library/Logs/scan                                                    |
build   19-Dec-2016 17:07:19    | open_report             | false                                                                  |
build   19-Dec-2016 17:07:19    | slack_only_on_failure   | false                                                                  |
build   19-Dec-2016 17:07:19    | use_clang_report_name   | false                                                                  |
build   19-Dec-2016 17:07:19    | fail_build              | true                                                                   |
build   19-Dec-2016 17:07:19    | xcode_path              | /Applications/Xcode.app                                                |
build   19-Dec-2016 17:07:19    +-------------------------+------------------------------------------------------------------------+
build   19-Dec-2016 17:07:19    
build   19-Dec-2016 17:07:19    [17:07:19]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme UnitTests -project ./myapp.xcodeproj -configuration Release -destination 'platform=iOS Simulator,id=4BFE000A-04CD-4B69-82BB-E4B61DDDB2FC' -derivedDataPath '/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/DerivedData/myapp' -enableCodeCoverage YES clean test | tee '/Users/me/Library/Logs/scan/MyApp-UnitTests.log' | xcpretty 
build   19-Dec-2016 17:07:19    [17:07:19]: ? Loading...
build   19-Dec-2016 17:07:20    [17:07:20]: ? Cleaning myapp/MyAppModel [Release]
build   19-Dec-2016 17:07:20    [17:07:20]: ? Check Dependencies
build   19-Dec-2016 17:07:20    [17:07:20]: ? Cleaning myapp/MyAppViewModel [Release]
build   19-Dec-2016 17:07:20    [17:07:20]: ? Check Dependencies
build   19-Dec-2016 17:07:20    [17:07:20]: ? Cleaning myapp/MyAppView [Release]
build   19-Dec-2016 17:07:20    [17:07:20]: ? Check Dependencies
build   19-Dec-2016 17:07:20    [17:07:20]: ? Cleaning myapp/MyApp [Release]
build   19-Dec-2016 17:07:20    [17:07:20]: ? Check Dependencies
build   19-Dec-2016 17:07:20    [17:07:20]: ? /var/folders/ys/szvd1t697xn8vf1g4g8dwpyw0000gr/T/SchemeScriptAction-pxYJrZ: line 2: : command not found
build   19-Dec-2016 17:07:20    [17:07:20]: ? Clean Succeeded
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.270 xcodebuild[14132:9247730]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
build   19-Dec-2016 17:07:20    [17:07:20]: ? /Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/DerivedData/myapp/Logs/Test/5801C679-4BE1-4E39-9D0E-902383857DC4/Session-MyAppViewModelTests-2016-12-19_170720-CjjTwn.log
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.271 xcodebuild[14132:9247678] [MT] IDETestOperationsObserverDebug: (41CBE38C-701A-42F7-AD2A-3BC328D40E58) Beginning test session MyAppViewModelTests-41CBE38C-701A-42F7-AD2A-3BC328D40E58 at 2016-12-19 17:07:20.270 with Xcode 8A218a on target <DVTiPhoneSimulator: 0x7fa3fc276500> {
build   19-Dec-2016 17:07:20    [17:07:20]: ? SimDevice: SimDevice : iPad 2 (4BFE000A-04CD-4B69-82BB-E4B61DDDB2FC) : state={ Booted } deviceType={ SimDeviceType : com.apple.CoreSimulator.SimDeviceType.iPad-2 } runtime={ SimRuntime : 9.3 (13E233) - com.apple.CoreSimulator.SimRuntime.iOS-9-3 }
build   19-Dec-2016 17:07:20    [17:07:20]: ? } (9.3 (13E233))
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.299 xcodebuild[14132:9247678] [MT] IDETestOperationsObserverDebug: (E08695C2-DAE1-4B36-8584-B3CB196B71A0) Beginning test session MyAppModelTests-E08695C2-DAE1-4B36-8584-B3CB196B71A0 at 2016-12-19 17:07:20.299 with Xcode 8A218a on target <DVTiPhoneSimulator: 0x7fa3fc276500> {
build   19-Dec-2016 17:07:20    [17:07:20]: ? SimDevice: SimDevice : iPad 2 (4BFE000A-04CD-4B69-82BB-E4B61DDDB2FC) : state={ Booted } deviceType={ SimDeviceType : com.apple.CoreSimulator.SimDeviceType.iPad-2 } runtime={ SimRuntime : 9.3 (13E233) - com.apple.CoreSimulator.SimRuntime.iOS-9-3 }
build   19-Dec-2016 17:07:20    [17:07:20]: ? } (9.3 (13E233))
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.300 xcodebuild[14132:9247738]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
build   19-Dec-2016 17:07:20    [17:07:20]: ? /Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/DerivedData/myapp/Logs/Test/5801C679-4BE1-4E39-9D0E-902383857DC4/Session-MyAppModelTests-2016-12-19_170720-xKf4in.log
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.322 xcodebuild[14132:9247678] [MT] IDETestOperationsObserverDebug: (6FE34F0A-558D-451D-941A-9AA0EC82DE06) Beginning test session MyAppViewTests-6FE34F0A-558D-451D-941A-9AA0EC82DE06 at 2016-12-19 17:07:20.322 with Xcode 8A218a on target <DVTiPhoneSimulator: 0x7fa3fc276500> {
build   19-Dec-2016 17:07:20    [17:07:20]: ? SimDevice: SimDevice : iPad 2 (4BFE000A-04CD-4B69-82BB-E4B61DDDB2FC) : state={ Booted } deviceType={ SimDeviceType : com.apple.CoreSimulator.SimDeviceType.iPad-2 } runtime={ SimRuntime : 9.3 (13E233) - com.apple.CoreSimulator.SimRuntime.iOS-9-3 }
build   19-Dec-2016 17:07:20    [17:07:20]: ? } (9.3 (13E233))
build   19-Dec-2016 17:07:20    [17:07:20]: ? 2016-12-19 17:07:20.322 xcodebuild[14132:9247790]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
build   19-Dec-2016 17:07:20    [17:07:20]: ? /Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/DerivedData/myapp/Logs/Test/5801C679-4BE1-4E39-9D0E-902383857DC4/Session-MyAppViewTests-2016-12-19_170720-p3eDoc.log
build   19-Dec-2016 17:07:20    [17:07:20]: ? Building myapp/MyAppModel [Release]
build   19-Dec-2016 17:07:20    [17:07:20]: ? Check Dependencies
build   19-Dec-2016 17:07:20    [17:07:20]: ? Processing Info.plist
build   19-Dec-2016 17:07:30    [17:07:30]: ? Compiling MyAppModel_vers.c
build   19-Dec-2016 17:07:30    [17:07:30]: ? Compiling MyAppModel_vers.c
build   19-Dec-2016 17:07:30    [17:07:30]: ? Linking MyAppModel
build   19-Dec-2016 17:07:30    [17:07:30]: ? Linking MyAppModel
build   19-Dec-2016 17:07:30    [17:07:30]: Running Tests: ? Touching MyAppModel.framework
build   19-Dec-2016 17:07:30    [17:07:30]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppModel.framework
build   19-Dec-2016 17:07:30    [17:07:30]: ? Building myapp/MyAppModelTests [Release]
build   19-Dec-2016 17:07:30    [17:07:30]: ? Check Dependencies
build   19-Dec-2016 17:07:30    [17:07:30]: ? Processing Info.plist
build   19-Dec-2016 17:07:48    [17:07:48]: ? Compiling MyAppModelTests_vers.c
build   19-Dec-2016 17:07:48    [17:07:48]: ? Compiling MyAppModelTests_vers.c
build   19-Dec-2016 17:07:48    [17:07:48]: ? Linking MyAppModelTests
build   19-Dec-2016 17:07:48    [17:07:48]: ? Linking MyAppModelTests
build   19-Dec-2016 17:08:08    [17:08:08]: ? Copying Carthage/Build/iOS/Quick.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Copying Carthage/Build/iOS/Haneke.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Copying Carthage/Build/iOS/Nimble.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Copying Carthage/Build/iOS/Alamofire.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppModelTests.xctest/Frameworks/Haneke.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppModelTests.xctest/Frameworks/Alamofire.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppModelTests.xctest/Frameworks/Quick.framework
build   19-Dec-2016 17:08:08    [17:08:08]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppModelTests.xctest/Frameworks/Nimble.framework
build   19-Dec-2016 17:08:08    [17:08:08]: Running Tests: ? Touching MyAppModelTests.xctest
build   19-Dec-2016 17:08:08    [17:08:08]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppModelTests.xctest
build   19-Dec-2016 17:08:09    [17:08:09]: ? Building myapp/MyAppViewModel [Release]
build   19-Dec-2016 17:08:09    [17:08:09]: ? Check Dependencies
build   19-Dec-2016 17:08:09    [17:08:09]: ? Processing Info.plist
build   19-Dec-2016 17:08:09    [17:08:09]: ? Compiling MyAppViewModel_vers.c
build   19-Dec-2016 17:08:09    [17:08:09]: ? Compiling MyAppViewModel_vers.c
build   19-Dec-2016 17:08:09    [17:08:09]: ? Linking MyAppViewModel
build   19-Dec-2016 17:08:09    [17:08:09]: ? Linking MyAppViewModel
build   19-Dec-2016 17:08:09    [17:08:09]: Running Tests: ? Touching MyAppViewModel.framework
build   19-Dec-2016 17:08:09    [17:08:09]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppViewModel.framework
build   19-Dec-2016 17:08:09    [17:08:09]: ? Building myapp/MyAppView [Release]
build   19-Dec-2016 17:08:09    [17:08:09]: ? Check Dependencies
build   19-Dec-2016 17:08:09    [17:08:09]: ? Compiling MyAppView_vers.c
build   19-Dec-2016 17:08:09    [17:08:09]: ? Compiling MyAppView_vers.c
build   19-Dec-2016 17:08:09    [17:08:09]: ? Linking MyAppView
build   19-Dec-2016 17:08:09    [17:08:09]: ? Linking MyAppView
build   19-Dec-2016 17:08:09    [17:08:09]: ? Processing Info.plist
build   19-Dec-2016 17:08:09    [17:08:09]: Running Tests: ? Touching MyAppView.framework
build   19-Dec-2016 17:08:09    [17:08:09]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppView.framework
build   19-Dec-2016 17:08:09    [17:08:09]: ? Building myapp/MyApp [Release]
build   19-Dec-2016 17:08:09    [17:08:09]: ? Check Dependencies
build   19-Dec-2016 17:08:11    [17:08:11]: ? Compiling MyApp_vers.c
build   19-Dec-2016 17:08:11    [17:08:11]: ? Compiling MyApp_vers.c
build   19-Dec-2016 17:08:19    [17:08:19]: ? Linking MyApp
build   19-Dec-2016 17:08:19    [17:08:19]: ? Linking MyApp
build   19-Dec-2016 17:08:19    [17:08:19]: ? Copying Settings.plist
build   19-Dec-2016 17:08:19    [17:08:19]: ? Processing Info.plist
build   19-Dec-2016 17:08:20    [17:08:20]: ? Copying DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppModel.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Copying DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppViewModel.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Copying DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyAppView.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/Frameworks/MyAppViewModel.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/Frameworks/MyAppModel.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/Frameworks/MyAppView.framework
build   19-Dec-2016 17:08:20    [17:08:20]: ? Running script 'Copy Carthage Frameworks'
build   19-Dec-2016 17:08:29    [17:08:29]: ? Running script 'SwiftLint'
build   19-Dec-2016 17:08:34    [17:08:34]: ? Running script 'Run fabric crashlytics'
build   19-Dec-2016 17:08:35    [17:08:35]: Running Tests: ? Touching MyApp.app
build   19-Dec-2016 17:08:35    [17:08:35]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app
build   19-Dec-2016 17:08:36    [17:08:36]: ? Building myapp/MyAppViewModelTests [Release]
build   19-Dec-2016 17:08:36    [17:08:36]: ? Check Dependencies
build   19-Dec-2016 17:08:36    [17:08:36]: ? Processing Info.plist
build   19-Dec-2016 17:08:36    [17:08:36]: ? Compiling MyAppViewModelTests_vers.c
build   19-Dec-2016 17:08:36    [17:08:36]: ? Compiling MyAppViewModelTests_vers.c
build   19-Dec-2016 17:08:36    [17:08:36]: ? Linking MyAppViewModelTests
build   19-Dec-2016 17:08:36    [17:08:36]: ? Linking MyAppViewModelTests
build   19-Dec-2016 17:08:36    [17:08:36]: ? Copying Carthage/Build/iOS/Alamofire.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Copying Carthage/Build/iOS/Quick.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Copying Carthage/Build/iOS/Nimble.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewModelTests.xctest/Frameworks/Alamofire.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewModelTests.xctest/Frameworks/Quick.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewModelTests.xctest/Frameworks/Nimble.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? /var/folders/ys/szvd1t697xn8vf1g4g8dwpyw0000gr/T/SchemeScriptAction-oHFvuL: line 2: : command not found
build   19-Dec-2016 17:08:36    [17:08:36]: Running Tests: ? Touching MyAppViewModelTests.xctest
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewModelTests.xctest
build   19-Dec-2016 17:08:36    [17:08:36]: ? Building myapp/MyAppViewTests [Release]
build   19-Dec-2016 17:08:36    [17:08:36]: ? Check Dependencies
build   19-Dec-2016 17:08:36    [17:08:36]: ? Compiling MyAppViewTests_vers.c
build   19-Dec-2016 17:08:36    [17:08:36]: ? Compiling MyAppViewTests_vers.c
build   19-Dec-2016 17:08:36    [17:08:36]: ? Linking MyAppViewTests
build   19-Dec-2016 17:08:36    [17:08:36]: ? Linking MyAppViewTests
build   19-Dec-2016 17:08:36    [17:08:36]: ? Processing Info.plist
build   19-Dec-2016 17:08:36    [17:08:36]: ? Copying Carthage/Build/iOS/Quick.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Copying Carthage/Build/iOS/Nimble.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewTests.xctest/Frameworks/Nimble.framework
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewTests.xctest/Frameworks/Quick.framework
build   19-Dec-2016 17:08:36    [17:08:36]: Running Tests: ? Touching MyAppViewTests.xctest
build   19-Dec-2016 17:08:36    [17:08:36]: ? Signing DerivedData/myapp/Build/Intermediates/CodeCoverage/Products/Release-iphonesimulator/MyApp.app/PlugIns/MyAppViewTests.xctest
build   19-Dec-2016 17:08:40    [17:08:40]: ? All tests
build   19-Dec-2016 17:08:40    [17:08:40]: ? Quick.framework
build   19-Dec-2016 17:08:40    [17:08:40]: ? Test Suite MyAppViewModelTests.xctest started
build   19-Dec-2016 17:08:40    [17:08:40]: ? MyPageViewModelSpec
build   19-Dec-2016 17:08:40    [17:08:40]: ? ? active_instances_returned_from_service_on_construction (0.039 seconds)
...
build   19-Dec-2016 17:08:41    [17:08:41]: ?    Executed 70 tests, with 0 failures (0 unexpected) in 0.156 (0.211) seconds
build   19-Dec-2016 17:08:41    [17:08:41]: ? 
build   19-Dec-2016 17:08:44    [17:08:44]: ? All tests
build   19-Dec-2016 17:08:44    [17:08:44]: ? Quick.framework
build   19-Dec-2016 17:08:44    [17:08:44]: ? Test Suite MyAppModelTests.xctest started
build   19-Dec-2016 17:08:44    [17:08:44]: ? MyPageServiceSpec
build   19-Dec-2016 17:08:44    [17:08:44]: ? ? service_specs__active_instances_are_correctly_returned (0.007 seconds)
...
build   19-Dec-2016 17:08:46    [17:08:46]: ?    Executed 144 tests, with 0 failures (0 unexpected) in 1.288 (1.401) seconds
build   19-Dec-2016 17:08:46    [17:08:46]: ? 
build   19-Dec-2016 17:08:49    [17:08:49]: ? All tests
build   19-Dec-2016 17:08:49    [17:08:49]: ? Test Suite MyAppViewTests.xctest started
build   19-Dec-2016 17:08:50    [17:08:50]: ? HtmlLabelSpec
build   19-Dec-2016 17:08:50    [17:08:50]: ? ? HtmlLabel__stringFrom_html__works_with_special_characters (0.005 seconds)
...
build   19-Dec-2016 17:08:50    [17:08:50]: ?    Executed 92 tests, with 0 failures (0 unexpected) in 0.457 (0.538) seconds
build   19-Dec-2016 17:08:50    [17:08:50]: ? 
build   19-Dec-2016 17:08:50    [17:08:50]: ? Quick.framework
build   19-Dec-2016 17:08:51    [17:08:51]: ? Test Succeeded
build   19-Dec-2016 17:08:52    +--------------------+-----+
build   19-Dec-2016 17:08:52    |       Test Results       |
build   19-Dec-2016 17:08:52    +--------------------+-----+
build   19-Dec-2016 17:08:52    | Number of tests    | 269 |
build   19-Dec-2016 17:08:52    | Number of failures | 0   |
build   19-Dec-2016 17:08:52    +--------------------+-----+
build   19-Dec-2016 17:08:52    
build   19-Dec-2016 17:08:53    [17:08:53]: Successfully generated report at '/Users/me/bamboo-agent-home/xml-data/build-dir/MYBUILD-JOB1/build/reports/junit_unittests.xml'
build   19-Dec-2016 17:08:53    
build   19-Dec-2016 17:08:53    +------+-------------------------------------+-------------+
build   19-Dec-2016 17:08:53    |                     fastlane summary                     |
build   19-Dec-2016 17:08:53    +------+-------------------------------------+-------------+
build   19-Dec-2016 17:08:53    | Step | Action                              | Time (in s) |
build   19-Dec-2016 17:08:53    +------+-------------------------------------+-------------+
build   19-Dec-2016 17:08:53    | 1    | Verifying required fastlane version | 0           |
build   19-Dec-2016 17:08:53    | 2    | default_platform                    | 0           |
build   19-Dec-2016 17:08:53    | 3    | scan                                | 96          |
build   19-Dec-2016 17:08:53    +------+-------------------------------------+-------------+
build   19-Dec-2016 17:08:53    
build   19-Dec-2016 17:08:53    [17:08:53]: fastlane.tools finished successfully ?
simple  19-Dec-2016 17:08:54    Finished task 'test app' with result: Success
command 19-Dec-2016 17:08:54    Substituting variable: ${bamboo.BUILD_CHANGE_LOG_PATH} with CHANGELOG.md
command 19-Dec-2016 17:08:54    Substituting variable: ${bamboo.JUnit_TestsPath} with build/reports/*.xml
picture garie

Most helpful comment

....
build   19-Dec-2016 17:08:41    [17:08:41]: ? �[35m�[32;1m   Executed 70 tests, with 0 failures (0 unexpected) in 0.156 (0.211) seconds�[0m
....
build   19-Dec-2016 17:08:46    [17:08:46]: ? �[35m�[32;1m   Executed 144 tests, with 0 failures (0 unexpected) in 1.288 (1.401) seconds�[0m
....
build   19-Dec-2016 17:08:46    [17:08:46]: ? �[35m�[32;1m   Executed 92 tests, with 0 failures (0 unexpected) in 1.288 (1.401) seconds�[0m

70 + 144 + 92 = 306

It seems like this is a bug of fastlane plugin, which scanning wrong number of tests.

picture wongzigii on 21 Dec 2016
❤3 🎉2

All 11 comments

This is probably a Quick issue, not a Nimble issue. Nimble just adds nicer test matchers/assertions.

@istx25 does this look familiar?

morganchen12 picture morganchen12 on 20 Dec 2016

Hmm. Not sure what's happening but it doesn't look like it would be an issue with Nimble. I'll have to do some more investigating.

noisypigeon picture noisypigeon on 20 Dec 2016

Okay. I wasn't sure where to start. I have no idea at which point the tests are disappearing. Let me know anything I can do to help!

garie picture garie on 20 Dec 2016

Mind refiling the issue on Quick? More traffic on there.

noisypigeon picture noisypigeon on 20 Dec 2016

Does your stack use xctool or something like that? It looks like the test output is being piped through some sort of formatter. Does your stack have a --verbose option that can display the full, unadulterated output from the tests?

modocache picture modocache on 20 Dec 2016 
....
build   19-Dec-2016 17:08:41    [17:08:41]: ? �[35m�[32;1m   Executed 70 tests, with 0 failures (0 unexpected) in 0.156 (0.211) seconds�[0m
....
build   19-Dec-2016 17:08:46    [17:08:46]: ? �[35m�[32;1m   Executed 144 tests, with 0 failures (0 unexpected) in 1.288 (1.401) seconds�[0m
....
build   19-Dec-2016 17:08:46    [17:08:46]: ? �[35m�[32;1m   Executed 92 tests, with 0 failures (0 unexpected) in 1.288 (1.401) seconds�[0m

70 + 144 + 92 = 306

It seems like this is a bug of fastlane plugin, which scanning wrong number of tests.

wongzigii picture wongzigii on 21 Dec 2016
❤3 🎉2

Wow, good catch!!

modocache picture modocache on 21 Dec 2016

@wongzigii Thanks so much! I will refile with fastlane.

garie picture garie on 22 Dec 2016

I've had a bit more time to run through this and figure out exactly what the problem is.

I can reproduce this with asynchronous tests. I have a test that uses waitUntil which does not show up in my xcodebuild log, does not report to the command line (unless it fails), and is not counted in this number. I believe that fastlane isn't counting it in the number because it uses xcpretty to parse the xcodebuild log and there is no output for this async test.

Unfortunately, while it is reproducible, it's not consistent. The same tests sometimes pass and appear in the output/count.

garie picture garie on 13 Feb 2017

I converted one of my tests from using waitUntil to toEventually in case the waitUntil was the problem but the test still is sometimes missing from the list.

garie picture garie on 13 Feb 2017

On second thought, what I said doesn't make sense. The test NEVER goes to the xcodebuild log but SOMETIMES is counted. I must be missing something else.

garie picture garie on 13 Feb 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

RonanMcH picture RonanMcH  ·  7Comments
nikolaykasyanov picture nikolaykasyanov  ·  9Comments
kleiberjp picture kleiberjp  ·  4Comments
zachlucas picture zachlucas  ·  5Comments
iby picture iby  ·  3Comments