Appcenter: Xamarin.UITest support fo iOS 13 and Xcode 11

[GeorgeDarakchiev]

I would like to be able to run UI tests on device\simulator running iOS13 (coming with Xcode 11).

[tjsomasundaram]

Hi, iOS 13 devices are currently selectable for testing. If you use Xamarin.UITest, you may need to update to Xcode 11 and the latest dev release of Xamarin.UITest. Please let us know if you attempt to use the Test service and are unable to select iOS 13 devices.

[Oddj0b]

@GeorgeDarakchiev Just to clarify is this locally or in App Center Test. Our engineers are still working on iOS 13 support for App Center Test as there are some unexpected challenges.

[j-hat]

@Oddj0b

Hey, running UI tests on any of the new iOS 13 simulators doesn't work for me since upgrading to Xcode 11.1

I get this error mentioning the iPhone 6 simulator and containing an absolute file path for a developer named Ivan. This same error was talked about here for building a Xamarin.iOS project, however for me this problem only occurs for running UI tests on Xamarin.iOS, building and deploying the app to the new simulators is working fine. The fix they recommended on that thread doesn't fix the UI tests.

Xamarin.UITest.XDB.Exceptions.DeviceAgentException : Failed to launch simulator

ExitCode: 134
    0   CoreFoundation                      0x00007fff4237e2fd __exceptionPreprocess + 256
    1   libobjc.A.dylib                     0x00007fff6ca78a17 objc_exception_throw + 48
    10  FBSimulatorControl                  0x0000000109505457 +[FBSimulator fromSimDevice:configuration:launchdSimProcess:containerApplicationProcess:set:] + 311
    11  FBSimulatorControl                  0x0000000109508aad -[FBSimulatorInflationStrategy inflateSimulators:availableDevices:previouslyIdentifiedContainerApplications:] + 1085
    12  FBSimulatorControl                  0x0000000109508588 -[FBSimulatorInflationStrategy inflateFromDevices:exitingSimulators:] + 1208
    13  FBSimulatorControl                  0x00000001094cd6f1 -[FBSimulatorSet allSimulators] + 145
    14  FBSimulatorControl                  0x00000001094caf8d -[FBSimulatorSet query:] + 173
    15  iOSDeviceManager                    0x00000001092a5e8b +[Simulator withID:] + 246
    16  iOSDeviceManager                    0x000000010929e300 +[Device withID:] + 118
    17  iOSDeviceManager                    0x00000001092a172b +[Command simulatorFromArgs:] + 163
    18  iOSDeviceManager                    0x00000001092adacc +[LaunchSimulatorCommand execute:] + 20
    19  iOSDeviceManager                    0x00000001092a4171 +[CLI process:] + 1111
    2   CoreFoundation                      0x00007fff42399016 +[NSException raise:format:arguments:] + 98
    20  iOSDeviceManager                    0x000000010928f539 main + 121
    21  libdyld.dylib                       0x00007fff6e2463d5 start + 1
    3   Foundation                          0x00007fff44638791 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 194
    4   FBSimulatorControl                  0x00000001094c561e +[FBSimulatorConfiguration makeDefaultConfiguration] + 558
    5   FBSimulatorControl                  0x00000001094c53a4 __48+[FBSimulatorConfiguration defaultConfiguration]_block_invoke + 36
    6   libdispatch.dylib                   0x00007fff6e1f963d _dispatch_client_callout + 8
    7   libdispatch.dylib                   0x00007fff6e1fad4b _dispatch_once_callout + 20
    8   FBSimulatorControl                  0x00000001094c5358 +[FBSimulatorConfiguration defaultConfiguration] + 136
    9   FBSimulatorControl                  0x000000010950beb0 +[FBSimulatorConfiguration(CoreSimulator) inferSimulatorConfigurationFromDevice:error:] + 704
(
)
*** First throw call stack:
2019-10-09 12:01:54.629 iOSDeviceManager[16737:439713] *** Assertion failure in +[FBSimulatorConfiguration makeDefaultConfiguration], /Users/Ivan/Documents/repos/Xamarin.UITest/FBSimulatorControl/FBSimulatorControl/Configuration/FBSimulatorConfiguration.m:65
2019-10-09 12:01:54.630 iOSDeviceManager[16737:439713] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Could not obtain OS for Default Device 'Model 'iPhone 6''. Available OS Versions []'
libc++abi.dylib: terminating with uncaught exception of type NSException

[King-of-Spades]

Version 3.0.4-dev2 has an update for local simulator tests using Xcode 11: https://www.nuget.org/packages/Xamarin.UITest/3.0.4-dev2

It might be worth giving it a shot @j-hat, though in the past, Xcode's minor version bumps (11.0 vs. 11.1, in this case) have sometimes required additional updates to Xamarin.UITest and/or the Test Cloud Agent; because despite being minor version bumps they contain changes which are breaking the test flow somehow.

[Oddj0b]

@GeorgeDarakchiev @j-hat does it work on the dev branch?

[GeorgeDarakchiev]

@Oddj0b we are running the tests locally and I can confirm that version 3.0.4-dev2 works with Xcode 11.0. and Xcode 11.1 and I am not getting the error @j-hat reported (I was not able to create iPhone 6 with iOS 13 simulator, but I have tested it on iPhone 11, iOS 13.1).

Sadly, after upgrading to 3.0.4-dev2 (from 3.0.0-dev1) I am getting the following exception on Android (running on Windows, not tested on Mac):

"Cannot access a disposed object.\r\nObject name: 'System.Net.Http.StringContent'."
at Xamarin.UITest.Shared.Http.HttpClient.SendData(String endpoint, String method, HttpContent content, ExceptionPolicy exceptionPolicy, Nullable`1 timeOut)
   at Xamarin.UITest.Shared.Http.HttpClient.PostUntilExpectedResultReceived(String endpoint, String arguments, String expectedContent, Int32 retries)
   at Xamarin.UITest.Shared.Android.HttpApplicationStarter.Execute(String intentJson)
   at Xamarin.UITest.Shared.Android.AndroidAppLifeCycle.LaunchApp(String appPackageName, ApkFile testServerApkFile, Int32 testServerPort)
   at Xamarin.UITest.Android.AndroidApp..ctor(IAndroidAppConfiguration appConfiguration, IExecutor executor)
   at Xamarin.UITest.Configuration.AndroidAppConfigurator.StartApp(AppDataMode appDataMode)
   at TestInitialize.TestInitializer.StartApp()

This can be reproduce by starting the app few times (in my measurements the crash occurs around the 12th start of the app, but it is not always the same):

                    for (int i = 0; i < 25; i++)
                    {
                        try
                        {
                            currentApp = appconfig.StartApp();
                        }
                        catch (Exception e)
                        {
                            Thread.Sleep(1000);
                        }
                    }

Here is my appconfig:

 var appconfig = ConfigureApp
                     .Android
                     .ApkFile(apk)
                     .EnableLocalScreenshots()
                     .DeviceSerial(currentDevice.Serial);

P.S. I am sorry for the late response.

[gziebold]

Just a FYI...using xcode 11.1 and the newish Version 3.0.4-dev2 I still get:

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Could not obtain OS for Default Device 'Model 'iPhone 6''. Available OS Versions []'

[olenka93]

Can't start any test with new version of Xcode 11.1 and Visual Studio for mac 8.3.4, the problem is still reproducible and really critical. DeviceAgent app could not be installed on simulator and tests fail with error
Exit code '134' received from '/var/folders/r9/bh9m7hpj3p1_nkj3lr06ly900000gq/T/xdb/DeviceAgent.iOS.Dependencies/4b3724d8104d6f340f7db14b4f047186/bin/iOSDeviceManager launch-simulator -d 7F65F7AC-37C9-45C5-8FCC-6579F03E01A6'
0 CoreFoundation 0x00007fff33698cfd __exceptionPreprocess + 256
1 libobjc.A.dylib 0x00007fff5dd60a17 objc_exception_throw + 48
10 FBSimulatorControl 0x0000000100b08457 +[FBSimulator fromSimDevice:configuration:launchdSimProcess:containerApplicationProcess:set:] + 311
11 FBSimulatorControl 0x0000000100b0baad -[FBSimulatorInflationStrategy inflateSimulators:availableDevices:previouslyIdentifiedContainerApplications:] + 1085
12 FBSimulatorControl 0x0000000100b0b588 -[FBSimulatorInflationStrategy inflateFromDevices:exitingSimulators:] + 1208
13 FBSimulatorControl 0x0000000100ad06f1 -[FBSimulatorSet allSimulators] + 145
14 FBSimulatorControl 0x0000000100acdf8d -[FBSimulatorSet query:] + 173
15 iOSDeviceManager 0x00000001008a4e7f +[Simulator withID:] + 246
16 iOSDeviceManager 0x000000010089d2f4 +[Device withID:] + 118
17 iOSDeviceManager 0x00000001008a071f +[Command simulatorFromArgs:] + 163
18 iOSDeviceManager 0x00000001008acac0 +[LaunchSimulatorCommand execute:] + 20
19 iOSDeviceManager 0x00000001008a3165 +[CLI process:] + 1111
2 CoreFoundation 0x00007fff336b3a1a +[NSException raise:format:arguments:] + 98
20 iOSDeviceManager 0x000000010088e52d main + 121
21 libdyld.dylib 0x00007fff5f52e3d5 start + 1
3 Foundation 0x00007fff35952c85 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 194
4 FBSimulatorControl 0x0000000100ac861e +[FBSimulatorConfiguration makeDefaultConfiguration] + 558
5 FBSimulatorControl 0x0000000100ac83a4 __48+[FBSimulatorConfiguration defaultConfiguration]_block_invoke + 36
6 libdispatch.dylib 0x00007fff5f4e163d _dispatch_client_callout + 8
7 libdispatch.dylib 0x00007fff5f4e2d4c _dispatch_once_callout + 20
8 FBSimulatorControl 0x0000000100ac8358 +[FBSimulatorConfiguration defaultConfiguration] + 136
9 FBSimulatorControl 0x0000000100b0eeb0 +[FBSimulatorConfiguration(CoreSimulator) inferSimulatorConfigurationFromDevice:error:] + 704
(
)
* First throw call stack:
2019-10-23 12:40:05.785 iOSDeviceManager[4716:138712] Assertion failure in +[FBSimulatorConfiguration makeDefaultConfiguration], /Users/Ivan/Documents/repos/Xamarin.UITest/FBSimulatorControl/FBSimulatorControl/Configuration/FBSimulatorConfiguration.m:65
2019-10-23 12:40:05.787 iOSDeviceManager[4716:138712] ** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Could not obtain OS for Default Device 'Model 'iPhone 6''. Available OS Versions []'
libc++abi.dylib: terminating with uncaught exception of type NSException

[jeromelaban]

Same issue on my end. Visual Studio 8.4 Preview, XCode 11.1 installed, running local tests.

[sebastianferrari]

Same issue here too. Any workaround available?

[gziebold]

@sebastianferrari my workaround is to have all my UI tests working is to have my xcode at v 11.0

[sebastianferrari]

Thank you @gziebold I'll downgrade it to v11.0. As it is now, it works fine on App Center Tests Platform.

[jdeff]

@gziebold I was also seeing issues on dev2. After updating to 3.0.4-dev4 everything seems to work for me with Xcode 11.1.

[King-of-Spades]

My teammate @ahanag22 has a theory to reproduce this issue, that might line up with the observations of those who can reproduce it vs. can't reproduce it:

I can reproduce the same with our sample. However, it was working fine just before I’ve updated the visual studio version to 8.4 Preview (8.4 build 1936). Previously, it was running fine on the latest stable version.

Can some of you who have reported that the dev version still doesn't work confirm your Visual Studio version? I'm not sure if we've confirmed that it's the VSMac version itself that causes the issue, or if it could be other tooling linked to that version / update channel. (Other common candidates are Mono & Xamarin.iOS, just because they're the most likely to have some type of interaction.)

[Oddj0b]

PSA: 3.0.5-dev5 has been released.

https://www.nuget.org/packages/Xamarin.UITest/3.0.4-dev5

[Oddj0b]

3.0.4 have been released! 🎆