Appcenter: Xamarin UI Test Not Supporting Xcode 12 & iOS 14

[Faraha87]

Recently updated Xamarin.iOS SDK to version 14.0.0.0 & Xcode 12.0 (12A7209) and are having issues running ReplConductor on Xamarin UI Tests.

Since updating to Xamarin.iOS 14, when starting a UI test using Xamarin.UITest (version 3.0.7) framework, the Device Agent on a physical device (iPhone) fails to launch resulting in ATP6.

Xamarin.UITest.XDB.Exceptions.DeviceAgentException : DeviceAgent start session failed: Failed to launch Device Agent

This had occurred on a previous version with Xcode 11 and iOS 13.
https://github.com/microsoft/appcenter/issues/1129

There has been no new Xamarin.UITest release version over the past 6 months
https://www.nuget.org/packages/Xamarin.UITest/3.0.7

When will they release a new version compatible with Xcode 12 and iOS 14?????

[dheerajnakhwa]

Exactly, what is happening? There has been no Xamarin UITest update since March 2020.

[Faraha87]

The same issue occurred in on the previous version with Xcode 11 & iOS 13 and it was fixed in Xamarin.UITest 3.0.7
https://www.nuget.org/packages/Xamarin.UITest/3.0.7

[DeerSteak]

AppCenter can't even build for iOS 14 yet based on the build options in my automated builds panel. See #2000

[magoolation]

Any news? We started a big project and iOS 14 is really a must have on our device matrix. Is there at least a roadmap?

[JohnH-FF]

There is another thread on this: https://github.com/microsoft/appcenter/issues/1992

Long story short, they are working on it and expect to have a release by the end of November.

[dimazaid]

This is released - please give it a try and let our support team know if you run into any issues.

[van8tka]

Hello, I use Xamarin.UITests 3.0.9, NUnit 3.12.0, Xamarin.TestCloud.Agent 0.22.1 and I have this below:

Xamarin.UITest.XDB.Exceptions.DeviceAgentException: Failed to launch DeviceAgent ExitCode: 65
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DeviceAgent-simulator.xctestrun -destination id=0916CD87-7E6C-4622-9B38-EA7D99434505 -derivedDataPath /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData
};
IDEDerivedDataPathOverride = /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData
IDETestRunSpecificationPath = /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DeviceAgent-simulator.xctestrun
XCSuiteRecordsKey = {
emitOSLogs:YES
testsToRun:(null)
testsToSkip:(null)
testBundleURL:file:///private/var/containers/Bundle/Application/21A83AF9-FACE-FACE-FACE-00605F990879/UITest-Runner.app/PlugIns/UITest.xctest
baselineFileURL:(null)
reportActivities:YES
testsDrivenByIDE:no
productModuleName:UITest
sessionIdentifier:(null)
reportResultsToIDE:no
targetApplicationPath:/Users/calabash/Library/Developer/Xcode/DerivedData/CBXDriver-thankyouforusingcalabash/Build/Products/Debug-iphoneos/TestApp.app
testExecutionOrdering:lexical
initializeForUITesting:YES
testBundleRelativePath:(null)
userAttachmentLifetime:0
automationFrameworkPath:(null)
baselineFileRelativePath:(null)
systemAttachmentLifetime:0
testsMustRunOnMainThread:YES
disablePerformanceMetrics:no
targetApplicationBundleID:sh.calaba.TestApp
targetApplicationArguments:(null)
randomExecutionOrderingSeed:(null)
gatherLocalizableStringsData:no
targetApplicationEnvironment:(null)
aggregateStatisticsBeforeCrash:
bridgedAutomationFrameworkPath:(null)
testApplicationDependencies:(null)
testApplicationUserOverrides:(null)
DeviceAgent-Runner.app (3257) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Crash: DeviceAgent-Runner (3257): . If you believe this error represents a bug, please attach the result bundle at /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData/Logs/Test/Test-Transient Testing-2020.11.19_10-34-51-+0300.xcresult))
SimDevice: iPhone 8 (0916CD87-7E6C-4622-9B38-EA7D99434505, iOS 13.0, Booted)
/var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData/Logs/Test/Test-Transient Testing-2020.11.19_10-34-51-+0300.xcresult
0 CoreFoundation 0x00007fff23b98bde __exceptionPreprocess + 350
1 libobjc.A.dylib 0x00007fff503b5b20 objc_exception_throw + 48
10 CoreFoundation 0x00007fff23af56b6 CFRunLoopRunSpecific + 438
11 GraphicsServices 0x00007fff3815cbb0 GSEventRunModal + 65
12 UIKitCore 0x00007fff47162a67 UIApplicationMain + 1621
13 DeviceAgent-Runner 0x000000010786f1f5 main + 183
14 libdyld.dylib 0x00007fff5123bcf5 start + 1
2 CoreFoundation 0x00007fff23b98958 +[NSException raise:format:arguments:] + 88
3 Foundation 0x00007fff255eb7be -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 166
4 XCTest 0x00000001079ad700 _XCTestMain + 761
5 DeviceAgent-Runner 0x000000010786f02d -[_XCTRunnerAppDelegate applicationWillResignActive:] + 0
6 DeviceAgent-Runner 0x000000010786ef2b _XCTRunnerRunTests + 0
7 CoreFoundation 0x00007fff23afb8ec __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
8 CoreFoundation 0x00007fff23afb058 __CFRunLoopDoBlocks + 312
9 CoreFoundation 0x00007fff23af5ee4 __CFRunLoopRun + 1284
TestTargetName:
treatMissingBaselinesAsFailures:no
(
)
* TEST EXECUTE FAILED *
* First throw call stack:
/var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData/Logs/Test/Test-Transient Testing-2020.11.19_10-34-51-+0300.xcresult/Staging/1_Test/Diagnostics/DeviceAgent-FDE5A303-42D2-41B2-AF03-D0478374B198/DeviceAgent-2717144D-EF65-4DAE-8420-95C764FB6035/Session-DeviceAgent-2020-11-19_103451-0H0wEb.log
{
}
} (13.0 (17A577))
2020-11-19 10:34:51.275 xcodebuild[3255:117691] [MT] IDETestOperationsObserverDebug: (EC2D8F10-7B15-408D-A4AE-6EEE850350CE) Beginning test session DeviceAgent-EC2D8F10-7B15-408D-A4AE-6EEE850350CE at 2020-11-19 10:34:51.276 with Xcode 12A7403 on target {
2020-11-19 10:34:51.275 xcodebuild[3255:117813] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
2020-11-19 10:34:51.282 xcodebuild[3255:117691] [MT] IDETestOperationsObserverDebug: (EC2D8F10-7B15-408D-A4AE-6EEE850350CE) Finished requesting crash reports. Continuing with testing.
2020-11-19 10:34:53.995 DeviceAgent-Runner[3257:117992] Running tests...
2020-11-19 10:34:53.997 DeviceAgent-Runner[3257:117992] Unable to load configuration data from specified path ; error: The file name is invalid.
2020-11-19 10:34:53.997 DeviceAgent-Runner[3257:117992] Unable to load configuration data from specified path ; error: The file name is invalid.
2020-11-19 10:34:53.999 DeviceAgent-Runner[3257:117992] Found most recent test bundle at NSBundle (not yet loaded)
2020-11-19 10:34:54.000 DeviceAgent-Runner[3257:117992] Found most recent test configuration in bundle: /Users/ivan/Library/Developer/CoreSimulator/Devices/0916CD87-7E6C-4622-9B38-EA7D99434505/data/Containers/Bundle/Application/0BF2C2F4-6B92-4D42-8650-8818894BFDD3/DeviceAgent-Runner.app/PlugIns/DeviceAgent.xctest/CBX.xctestconfiguration
2020-11-19 10:34:54.001 DeviceAgent-Runner[3257:117992] clearing out IDE related attributes
2020-11-19 10:34:54.001 DeviceAgent-Runner[3257:117992] Running with configuration
2020-11-19 10:34:54.007 DeviceAgent-Runner[3257:117992] Assertion failure in void _XCTestMain(XCTestConfiguration *__strong _Nonnull)(), /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-14460.20/Sources/XCTestFramework/Harness/XCTestMain.m:213
2020-11-19 10:34:54.009 DeviceAgent-Runner[3257:117992] ** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No test bundle found at file:///private/var/containers/Bundle/Application/21A83AF9-FACE-FACE-FACE-00605F990879/UITest-Runner.app/PlugIns/UITest.xctest'
2020-11-19 10:34:57.834 xcodebuild[3255:117691] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2020-11-19 10:34:57.834 xcodebuild[3255:117691] [MT] IDETestOperationsObserverDebug: 6.578 elapsed -- Testing started completed.
2020-11-19 10:34:57.834 xcodebuild[3255:117691] [MT] IDETestOperationsObserverDebug: 6.578 sec, +6.578 sec -- end
2020-11-19 10:34:59.599 xcodebuild[3255:117691] [MT] IDEResultKit: Warning: While writing a result bundle to /var/folders/09/g_1b92q928gbxcj3_1v5yf1c0000gn/T/xdb/logs/2020.11.19.103450/DerivedData/Logs/Test/Test-Transient Testing-2020.11.19_10-34-51-+0300.xcresult, not all contents have been imported, as the Staging directory still contains the following files/directories: ["1_Test"]
Command line invocation:
libc++abi.dylib: terminating with uncaught exception of type NSException
Test session results, code coverage, and logs:
Testing failed:
Testing started
User defaults from command line:
at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.LaunchTestAsync (System.String deviceId, System.String deviceAddress) [0x00353] in <1da92138055146bba74b17eb98358802>:0
at Xamarin.UITest.iOS.iOSAppLauncher.LaunchAppLocal (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, System.Boolean clearAppData) [0x001e8] in <1da92138055146bba74b17eb98358802>:0
at Xamarin.UITest.iOS.iOSAppLauncher.LaunchApp (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, Xamarin.UITest.TestCloud.TestCloudiOSAppConfiguration testCloudAppConfiguration, Xamarin.UITest.Shared.Http.HttpClient testCloudWsClient, Xamarin.UITest.Shared.Http.HttpClient xtcServicesClient, System.Boolean testCloudUseDeviceAgent) [0x00068] in <1da92138055146bba74b17eb98358802>:0
at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Execution.IExecutor executor) [0x0026f] in <1da92138055146bba74b17eb98358802>:0
at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration) [0x00000] in <1da92138055146bba74b17eb98358802>:0
at Xamarin.UITest.Configuration.iOSAppConfigurator.StartApp (Xamarin.UITest.Configuration.AppDataMode appDataMode) [0x00017] in <1da92138055146bba74b17eb98358802>:0
at HomeWork.iOS.UITests.Authorization_Tests.BeforeEachTest () [0x00002] in /Users/ivan/Documents/ios/trunk/HomeWork.iOS.UITests/Authorization_Tests.cs:36

[dheerajnakhwa]

I also got the same error when I tried to use Xamarin.UITests 3.0.9

[JohnH-FF]

@dimazaid We are still getting the error Failed to launch DeviceAgent ExitCode: 65 with Xamarin.UITest 3.0.9.
This is the same error that was reported 17th Sept in the thread you locked and closed.

I would be happy to volunteer to beta test a fix for this if that helps? But we need a fix regardless.

[JohnH-FF]

@dimazaid would you prefer we raise another issue to track this? Or will you reopen either of these 2 issues?

[dimazaid]

Sorry to hear that. I have notified the team. The best way to get support from our engineers would be to contact support button on appcenter.ms (top right corner -> "?" -> "contact support".

[JohnH-FF]

@dimazaid thanks, have just done that. Please keep us updated on progress!

[Cheesebaron]

@dimazaid what is the purpose of this issue tracker if we need to use "contact support" on the web page?

[lubiepomaranczki]

oh, c'mon! A new version was released and we experience the same issue as @van8tka... great! ❤️

I expect that if many people are having this issue we can expect a build with a fix. When will it happen @dimazaid?

[JohnH-FF]

I have been contacted by support and they are now investigating further. I suggest you all do the same pls, it will help raise the profile within the support team. I replicated the issue on local device as well as simulator before they informed the test team.

[yamamoo]

I use Xamarin.UITest locally and do not really use App Center, but I reported my problem in App Center.

[yurii-harkusha]

I have the same issue even after it supposed to be fixed. I see it for both emulator and physical device. Our team also reported it to MS support. They are investigating it now.

[arevellfraedom]

We also have the same issue running tests locally. Have submitted report to App Center support.

[DaphnaPorath]

hi @dimazaid,
I've opened a support call as you've requested.
Would appreciate your help promoting this issue, as it has been around for quite some time and it is very significant!
Is there an ETA for fixing this issue?

[seanrodda]

@dimazaid
I have contacted supported as I am also experiencing this issue. Please let us know of any update. Surely communication through this ticket or another similar would save everyone time, including your support team receiving new tickets for the same issue.

[RobertWildgoose]

We are having the same issue, Is there any further update, or a rough ETA?

[yamamoo]

I reported my problem in appsenter.ms three days ago and I was contacted by a support person. He asked me to provide the test run URL. Since I do not use App Center for running my test, I cannot provide the URL they asked for. I will try to explain, but I'm afraid my support request might be rejected as I'm unable to give them what they are asking. If anyone using App Center and having this problem, please contact support and provide your URL.

[JohnHDev]

@yamamoo they asked the same of me, I pointed out that we can't get as far as running it in AppCenter without being able to write the tests and run them locally first. You could mention that requirement when you get back to them.

[dimazaid]

I'm following up on this with the team. They're already looking into this. We Apologize for the inconvenience this may have caused.

[JohnH-FF]

BREAKING NEWS: I have had feedback from the test team that they know about the bug and are testing a fix for it today. I have volunteered to be a beta tester if needed!
So watch this space...

[JohnH-FF]

I have tested 3.0.10 and still experience the same errorcode 65 issue. I have reported back and they are investigating.

[jpc1963]

yes still happening with 3.0.10.

[arevellfraedom]

I also checked 3.0.10 to ensure there wasn't something contextual about success vs failure. But no luck for us either

[arevellfraedom]

I have just checked 3.0.11 and it appears to be working correctly in XCode 12 based on initial tests 🚀

[seanrodda]

3.0.11 also working for myself. Thanks team

[yamamoo]

Thanks for good news! I tested 3.0.11 and it worked as well.

[JohnHDev]

Confirmed, 3.0.11 ia working as expected for us too.

[mahesh-kumbhar]

Not working for me With Xcode12.2
TestCloud Agent : 0.22.1
Xamarin UI test : 3.0.11

Getting below Error
SetUp : Application failed to launch! Failed to launch simulator

ExitCode: 134
0 CoreFoundation 0x00007fff2e2bad07 __exceptionPreprocess + 250
1 libobjc.A.dylib 0x00007fff670255bf objc_exception_throw + 48
10 FBSimulatorControl 0x000000010b67a35f +[FBSimulator fromSimDevice:configuration:launchdSimProcess:containerApplicationProcess:set:] + 303
11 FBSimulatorControl 0x000000010b67d7da -[FBSimulatorInflationStrategy inflateSimulators:availableDevices:previouslyIdentifiedContainerApplications:] + 1082
12 FBSimulatorControl 0x000000010b67d2ba -[FBSimulatorInflationStrategy inflateFromDevices:exitingSimulators:] + 1194
13 FBSimulatorControl 0x000000010b64350a -[FBSimulatorSet allSimulators] + 138
14 FBSimulatorControl 0x000000010b640dc3 -[FBSimulatorSet query:] + 163
15 iOSDeviceManager 0x000000010b3de33f +[Simulator withID:] + 247
16 iOSDeviceManager 0x000000010b3d696c +[Device withID:] + 118
17 iOSDeviceManager 0x000000010b3d9cbe +[Command simulatorFromArgs:] + 163
18 iOSDeviceManager 0x000000010b3e5d38 +[LaunchSimulatorCommand execute:] + 20
19 iOSDeviceManager 0x000000010b3dc72f +[CLI process:] + 1101
2 CoreFoundation 0x00007fff2e2e3ec8 +[NSException raise:format:arguments:] + 88
20 iOSDeviceManager 0x000000010b3c8269 main + 121
21 libdyld.dylib 0x00007fff681cccc9 start + 1
22 ??? 0x0000000000000004 0x0 + 4
3 Foundation 0x00007fff309cef3d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 191
4 FBSimulatorControl 0x000000010b63b655 +[FBSimulatorConfiguration makeDefaultConfiguration] + 517
5 FBSimulatorControl 0x000000010b63b404 __48+[FBSimulatorConfiguration defaultConfiguration]_block_invoke + 36
6 libdispatch.dylib 0x00007fff68173658 _dispatch_client_callout + 8
7 libdispatch.dylib 0x00007fff681747de _dispatch_once_callout + 20
8 FBSimulatorControl 0x000000010b63b3b8 +[FBSimulatorConfiguration defaultConfiguration] + 136
9 FBSimulatorControl 0x000000010b680bc7 +[FBSimulatorConfiguration(CoreSimulator) inferSimulatorConfigurationFromDevice:error:] + 679
(
)
* First throw call stack:
2020-12-03 16:37:03.257 iOSDeviceManager[5238:124768] Assertion failure in +[FBSimulatorConfiguration makeDefaultConfiguration], /Users/admin/git/FBSimulatorControl/FBSimulatorControl/Configuration/FBSimulatorConfiguration.m:65
2020-12-03 16:37:03.258 iOSDeviceManager[5238:124768] ** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Could not obtain OS for Default Device 'Model 'iPhone 6''. Available OS Versions []'
libc++abi.dylib: terminating with uncaught exception of type NSException
TearDown : System.NullReferenceException : Object reference not set to an instance of an object

[DmitriiKrupinECM]

What I'm observing after the update is that Xamarin UI tests are running terribly slowly on iOS 14.2.
One simulator test takes up to 5 minutes (locally as well as in Azure pipeline builds) on 14.2 whereas testing on 13.5 with the same device simulator finishes in 1.5 minutes.

[JohnH-FF]

@DmitriiKrupinECM I suggest you open a new thread for this issue, or contact support directly. Closed threads rarely get responded to by MS.

[andrew-lundy]

We currently support iOS 14.

The most recent version of NuGet can be found here: https://www.nuget.org/packages/Xamarin.UITest/3.0.14.

@DmitriiKrupinECM please open a new issue if you're still experiencing what you described here: https://github.com/microsoft/appcenter/issues/1998#issuecomment-742382002.