Appcenter: NUnit 3 / UITest failing with error: Exception has been thrown by the target of an invocation

Created on 15 May 2019  路  49Comments  路  Source: microsoft/appcenter

What App Center service does this affect?
testing on ios simulators / android emulators

Describe the bug
Hi I have a Xamarin UI test project with tests apps on iOS and Android simulators. I have recently updated nunit & xamarin.uitest to the latest versions. Now when I start my app on a simulator I get the error: Exception has been thrown by the target of an invocation

To Reproduce
I start the app as follows: (the app is started on the simulator - but then the test crashes)

IApp myApp =  ConfigureApp
            .iOS
            .InstalledApp(pathToApp)
            .DeviceIdentifier(123456-123456)
            .PreferIdeSettings()
            .EnableLocalScreenshots()
            .WaitTimes(new WaitTimes())
            .Debug()
            .StartApp(ClearAppData);

Expected behavior
With the app now visible and running I expect to be able to test it. With the variable myApp I should be able to run my tests. But the error occurs in the ConfigureApp and the variable myApp remains null.

Screenshots
none

Desktop (please complete the following information):

  • OS: Mac OS 10.14.4
  • Visual Studio for Mac: 8.0.5

Additional context
Nuget packages:

  • Nunit: 3.11
  • NUnit console runner: 3.10
  • Specflow 3.0.213
  • Xamarin UITest 3.0.0
  • xCode 10.2.1

The following is the log output after trying to start an ios app:

1 - Exception
Exception: System.Threading.ThreadAbortException: Thread was being aborted.
at (wrapper managed-to-native) System.Threading.Monitor.Monitor_wait(object,int)
at System.Threading.Monitor.ObjWait (System.Boolean exitContext, System.Int32 millisecondsTimeout, System.Object obj) [0x0002f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.Threading/Monitor.cs:83
at System.Threading.Monitor.Wait (System.Object obj, System.Int32 millisecondsTimeout, System.Boolean exitContext) [0x0000e] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/monitor.cs:210
at System.Threading.Monitor.Wait (System.Object obj, System.Int32 millisecondsTimeout) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/monitor.cs:220
at System.Threading.ManualResetEventSlim.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00141] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/ManualResetEventSlim.cs:669
at System.Threading.Tasks.Task.WaitAllBlockingCore (System.Collections.Generic.LowLevelListWithIList 1[T] tasks, System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00038] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:4703
at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x000e6] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:4619
at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:4482
at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-08/external/bockbuild/builds/mono-x64/external/corert/src/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:4405
at Xamarin.UITest.XDB.Services.Processes.ProcessService.Run (System.String command, System.String arguments) [0x0004f] in <18ae7883e2424c558186d1d9edf9f14b>:0
at Xamarin.UITest.XDB.Services.OSX.XcodeService.TestWithoutBuilding (System.String deviceId, System.String xctestrunPath, System.String derivedDataPath) [0x0002b] in <18ae7883e2424c558186d1d9edf9f14b>:0
at Xamarin.UITest.XDB.Services.OSX.iOSDeviceManagerService.StartTest (System.String deviceId) [0x00022] in <18ae7883e2424c558186d1d9edf9f14b>:0
at Xamarin.UITest.XDB.Services.iOSDeviceAgentService+<>c__DisplayClass17_0.b__2 () [0x00000] in <18ae7883e2424c558186d1d9edf9f14b>:0
at Xamarin.UITest.XDB.Services.iOSDeviceAgentService.WithErrorHandling (System.Int32 eventId, System.Func 1[TResult] action, System.String errorMessage, System.Int32[] successCodes) [0x0000e] in <18ae7883e2424c558186d1d9edf9f14b>:0

bug test
Source
picture jpc1963
👍3

Most helpful comment

I am currently running my tests using NUnit 3.12, Xamarin.UITest 3.0.3
I no longer get the original error or the "connection refused" error

picture jpc1963 on 13 Aug 2019
👍4

All 49 comments

I'm also having this issue. Is there a known workaround for it ?

NUnit 3 is increasingly becoming a blocking feature, as NUnit 2 test runner support is very unstable in VS 2019.

jeromelaban picture jeromelaban on 21 Jun 2019

@jeromelaban @jpc1963 Are you getting the same error if you disable the local screenshots?
I'm mainly asking this because we already have an existing bug filed here https://github.com/microsoft/appcenter/issues/563, SpecFlow version 3 on App Center Test doesn't display per-step screenshot.

ahanag22 picture ahanag22 on 21 Jun 2019

The behavior is the same with or without local screenshots.

Also, it might be of interest that it seems that this exception appers on the unit tests side:

System.Net.Http.HttpRequestException : Connection refused
  ----> System.Net.Sockets.SocketException : Connection refused
TearDown : System.NullReferenceException : Object reference not set to an instance of an object
  at System.Net.Http.ConnectHelper.ConnectAsync (System.String host, System.Int32 port, System.Threading.CancellationToken cancellationToken) [0x001ac] in <d557f813f52a4dafa05dcf2b14802e7d>:0
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00134] in <d557f813f52a4dafa05dcf2b1                                     4802e7d>:0
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync (System.Threading.Tasks.ValueTask`1[TResult] creationTask) [0x000a2] in <d557f813f52a4dafa05dcf2b14802e7d>:0
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x00089] in                                      <d557f813f52a4dafa05dcf2b14802e7d>:0
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ba] in <d557f813f52a4dafa05dcf2b14802e7d>:0
  at Xamarin.UITest.Shared.Http.HttpClient.Request (System.String method, System.String endpoint, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x000f6] in                                      <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.Shared.Http.HttpClient.Get (System.String endpoint, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00000] in <18ae7883e2424c558186d1d9                                     edf9f14b>:0
  at Xamarin.UITest.iOS.HttpCalabashConnection.Version () [0x0001f] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.iOS.iOSAppLauncher.GetCalabashDevice (Xamarin.UITest.iOS.ICalabashConnection connection) [0x00075] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.iOS.iOSAppLauncher.EnsureCalabashRunning (Xamarin.UITest.iOS.ICalabashConnection connection) [0x00000] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.iOS.iOSAppLauncher.LaunchApp (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, Xamarin.UITest.TestCloud.TestClo                                     udiOSAppConfiguration testCloudAppConfiguration, Xamarin.UITest.Shared.Http.HttpClient testCloudWsClient, Xamarin.UITest.Shared.Http.HttpClient xtcServicesClient, System.Boolean testCloudUseDeviceAgen                                     t) [0x0007f] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Execution.IExecutor executor) [0x0028d] in <18ae7883e2424c558186d1d9edf9                                     f14b>:0
  at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration) [0x00000] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at Xamarin.UITest.Configuration.iOSAppConfigurator.StartApp (Xamarin.UITest.Configuration.AppDataMode appDataMode) [0x00017] in <18ae7883e2424c558186d1d9edf9f14b>:0
  at SamplesApp.UITests.AppInitializer.CreateiOSApp () [0x0003a] in <a01f077c2d344485a748a1ff1dd7b04f>:0
  at SamplesApp.UITests.AppInitializer.StartApp () [0x00056] in <a01f077c2d344485a748a1ff1dd7b04f>:0
  at Sample.UITests.TestBase.StartApp () [0x00000] in <a01f077c2d344485a748a1ff1dd7b04f>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.Culture                                     Info culture) [0x0006a] in <63a185bccd644d7ea7199ca475c500f2>:0
--SocketException
  at System.Net.Http.ConnectHelper.ConnectAsync (System.String host, System.Int32 port, System.Threading.CancellationToken cancellationToken) [0x00110] in <d557f813f52a4dafa05dcf2b14802e7d>:0
--TearDown
  at Sample.UITests.TestBase.CloseApp () [0x00006] in <a01f077c2d344485a748a1ff1dd7b04f>:0
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.Culture                                     Info culture) [0x0006a] in <63a185bccd644d7ea7199ca475c500f2>:0
jeromelaban picture jeromelaban on 21 Jun 2019

@jeromelaban Are you using Xamarin.TestCloud.Agent latest version 0.21.8 in your iOS app?

ahanag22 picture ahanag22 on 21 Jun 2019

Yes, exactly this version, and Xamarin.UITest 3.0.0.

jeromelaban picture jeromelaban on 21 Jun 2019

Thanks for confirming. I'll try to reproduce this behavior on my end. Meanwhile, can you try this sample app & UITest project?https://github.com/King-of-Spades/AppCenter-Samples/tree/master/Xamarin.UITest/UITestDemo/iOS -
please let me know if that project works for you? If the sample project doesn't work for you, does it hit the same error, or something else?

ahanag22 picture ahanag22 on 22 Jun 2019

The code I'm using is open source, so I can also provide you with a repro. I'll prepare that and test with that. Thanks!

jeromelaban picture jeromelaban on 22 Jun 2019

I've having the exact same issue with your sources. The simulator starts multiple times, then _the uitests_ fail while the app starts.

How can I help you troubleshoot that further ? I'm using VS4mac 8.1 Preview (build 2460)

jeromelaban picture jeromelaban on 22 Jun 2019

I get the System.Net.Http.HttpRequestException : Connection refused error running the tests in https://github.com/King-of-Spades/AppCenter-Samples/tree/master/Xamarin.UITest/UITestDemo/iOS when using VS for Mac 8.2 Preview (build 732). They work fine if I use the stable version of VS for Mac.

This will be investigated and someone will let you know when it's been solved but for now I would suggest using VS for Mac stable if possible.

stoneman picture stoneman on 24 Jun 2019
👍2

@jonstoneman Thanks for the followup! Unfortunately, I cannot do this, as there are specifics in the way VS4Mac has improved (particularly with regards to nuget support for cross-targeted projects) that enable key scenarios.

It was previously a very complex task for us to test applications that are built inside of a cross-targeted framework (those needed to be built with VS4Win) and those VS4Mac updates enable that. This is the last key piece that will make this UI Testing scenario possible for us.

Thanks!

jeromelaban picture jeromelaban on 25 Jun 2019

I am currently running VS for Mac 8.1.5 (Stable Channel) and am experiencing this as well. Is there a fix and/or workaround? Thanks!

adamzucchi picture adamzucchi on 22 Jul 2019
👍2

I've been poking at this issue again, and the latest 3.0.1-dev2 along with VS4Mac 8.2 (build 1090) is getting farther along, but is now giving errors such as this one:

Unable to scan assembly: nunit.engine, Version=3.9.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Mono.Cecil, Version=0.9.6.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756' or one of its dependencies.
File name: 'Mono.Cecil, Version=0.9.6.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756'
  at (wrapper managed-to-native) System.Reflection.MonoMethodInfo.get_method_info(intptr,System.Reflection.MonoMethodInfo&)
  at System.Reflection.MonoMethodInfo.GetMethodInfo (System.IntPtr handle) [0x00000] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Reflection.RuntimeMethodInfo.GetPseudoCustomAttributes () [0x00002] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.MonoCustomAttrs.GetPseudoCustomAttributes (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType) [0x0000c] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.MonoCustomAttrs.GetCustomAttributesBase (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inheritedOnly) [0x0001f] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.MonoCustomAttrs.GetCustomAttributes (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inherit) [0x00037] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Reflection.RuntimeMethodInfo.GetCustomAttributes (System.Type attributeType, System.Boolean inherit) [0x00000] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element, System.Type type, System.Boolean inherit) [0x00088] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Attribute.GetCustomAttributes (System.Reflection.MemberInfo element, System.Type type) [0x00000] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Reflection.CustomAttributeExtensions.GetCustomAttributes (System.Reflection.MemberInfo element, System.Type attributeType) [0x00000] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at Xamarin.UITest.Events.EventManager+<>c__DisplayClass2_0.<GetStaticMethodsWithAttribute>b__2 (<>f__AnonymousType3`2[<type>j__TPar,<method>j__TPar] <>h__TransparentIdentifier0) [0x00006] in <8139e2cb182043debeabb4fdea6d573a>:0
  at System.Linq.Enumerable+<SelectManyIterator>d__167`3[TSource,TCollection,TResult].MoveNext () [0x0004e] in <5b3a07cbef83409ea3183d1ea9f1b76a>:0
  at System.Collections.Generic.List`1[T].AddEnumerable (System.Collections.Generic.IEnumerable`1[T] enumerable) [0x00059] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x000f4] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <db6d3f68ee4f4e0ab4ffdf0911bd3e33>:0
  at Xamarin.UITest.Events.EventManager.GetStaticMethodsWithAttribute (System.Type attributeType) [0x000bb] in <8139e2cb182043debeabb4fdea6d573a>:0
jeromelaban picture jeromelaban on 23 Jul 2019

Also, now that VS 8.2 is out, it'll become critical for this to work.

jeromelaban picture jeromelaban on 24 Jul 2019

This seems to be related to the version of mono - if I change my mono version:
Screenshot 2019-07-25 at 13 07 53

Then UITest works correctly for me locally.

I'm not sure at what mono version. this started failing (5.18.1 was just a version I happened to have on my machine) but as a workaround for now I would try an earlier mono while we. investigate.

owenniblock picture owenniblock on 25 Jul 2019
👍3

@jeromelaban - we're aware of the Cecil issue. Hoping to have a new dev release out with a fix soon.

owenniblock picture owenniblock on 25 Jul 2019

@jeromelaban: UITest 3.0.2-dev1 has been released which should fix the Cecil issue: https://www.nuget.org/packages/Xamarin.UITest/3.0.2-dev1

stoneman picture stoneman on 25 Jul 2019

@jonstoneman thanks! Actually, the latest I mentioned uses 3.0.1-dev2, so the issue seems to still be present. Preview bits (dev1 included) do not work at all for me (connection issues), dev2 is the only one that goes farther along but fails for other reasons.

jeromelaban picture jeromelaban on 25 Jul 2019

I am experiencing a similar issue, including on Xamarin.UITest 3.0.2-dev1.

This exception only appears on Xamarin.iOS; Xamarin.Android is working fine.

Reproduction Sample

https://github.com/brminnick/InvestmentDataSampleApp

Stack Trace

at System.Net.Http.ConnectHelper.ConnectAsync (System.String host, System.Int32 port, System.Threading.CancellationToken cancellationToken) [0x00180] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:84
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ea] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:371
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync (System.Threading.Tasks.ValueTask1[TResult] creationTask) [0x000a2] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:529 at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x0003f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:284 at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00070] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:32 at Xamarin.UITest.Shared.Http.HttpClient.Request (System.String method, System.String endpoint, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable1[T] timeOut) [0x000f6] in :0
at Xamarin.UITest.Shared.Http.HttpClient.Get (System.String endpoint, Xamarin.UITest.Shared.Http.ExceptionPolicy exceptionPolicy, System.Nullable`1[T] timeOut) [0x00000] in :0
at Xamarin.UITest.iOS.HttpCalabashConnection.Version () [0x0001f] in :0
at Xamarin.UITest.iOS.iOSAppLauncher.GetCalabashDevice (Xamarin.UITest.iOS.ICalabashConnection connection) [0x00075] in :0
at Xamarin.UITest.iOS.iOSAppLauncher.EnsureCalabashRunning (Xamarin.UITest.iOS.ICalabashConnection connection) [0x00000] in :0
at Xamarin.UITest.iOS.iOSAppLauncher.LaunchApp (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Http.HttpClient httpClient, Xamarin.UITest.TestCloud.TestCloudiOSAppConfiguration testCloudAppConfiguration, Xamarin.UITest.Shared.Http.HttpClient testCloudWsClient, Xamarin.UITest.Shared.Http.HttpClient xtcServicesClient, System.Boolean testCloudUseDeviceAgent) [0x0007f] in :0
at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration, Xamarin.UITest.Shared.Execution.IExecutor executor) [0x0026f] in :0
at Xamarin.UITest.iOS.iOSApp..ctor (Xamarin.UITest.Configuration.IiOSAppConfiguration appConfiguration) [0x00000] in :0
at Xamarin.UITest.Configuration.iOSAppConfigurator.StartApp (Xamarin.UITest.Configuration.AppDataMode appDataMode) [0x00017] in :0
at InvestmentDataSampleApp.UITests.AppInitializer.StartApp (Xamarin.UITest.Platform platform) [0x0001e] in /Users/bramin/GitHub/InvestmentDataSampleApp/Src/InvestmentDataSampleApp.UITests/AppInitializer.cs:16
at InvestmentDataSampleApp.UITests.BaseTest.TestSetup () [0x00001] in /Users/bramin/GitHub/InvestmentDataSampleApp/Src/InvestmentDataSampleApp.UITests/Tests/BaseTest.cs:28
at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.Reflection/RuntimeMethodInfo.cs:391
--SocketException
at System.Net.Http.ConnectHelper.ConnectAsync (System.String host, System.Int32 port, System.Threading.CancellationToken cancellationToken) [0x000c8] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-02/external/bockbuild/builds/mono-x64/external/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs:65

Environment

=== Visual Studio Enterprise 2019 for Mac ===

Version 8.2 (build 1105)
Installation UUID: e8d39274-f49e-49c5-ac5a-5e89b4e131e6
GTK+ 2.24.23 (Raleigh theme)
Xamarin.Mac 5.14.0.85 (d16-2-fix-mac-updates / 86973979)

Package version: 600000311

=== Mono Framework MDK ===

Runtime:
Mono 6.0.0.311 (2019-02/494641b300c) (64-bit)
Package version: 600000311

=== NuGet ===

Version: 5.1.0.6013

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
3.0.0-preview7-27912-14
3.0.0-preview6-27804-01
2.1.11
SDK: /usr/local/share/dotnet/sdk/3.0.100-preview7-012821/Sdks
SDK Versions:
3.0.100-preview7-012821
3.0.100-preview6-012264
2.1.700
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/6.0.0/lib/mono/msbuild/Current/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.6.11
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Updater ===

Version: 11

=== Apple Developer Tools ===

Xcode 10.3 (14492.2)
Build 10G8

=== Xamarin.Mac ===

Xamarin.Mac not installed. Can't find /Library/Frameworks/Xamarin.Mac.framework/Versions/Current/Version.

=== Xamarin.iOS ===

Version: 12.14.0.110 (Visual Studio Enterprise)
Hash: a8bcecce
Branch: d16-2
Build date: 2019-06-28 07:31:42-0400

=== Xamarin Designer ===

Version: 16.2.0.354
Hash: 5f80a14f8
Branch: remotes/origin/d16-2
Build date: 2019-07-11 17:28:26 UTC

=== Xamarin.Android ===

Version: 9.4.0.51 (Visual Studio Enterprise)
Commit: xamarin-android/d16-2/9fa7775
Android SDK: /Users/bramin/Library/Developer/Xamarin/android-sdk-macosx
Supported Android versions:
None installed

SDK Tools Version: 26.1.1
SDK Platform Tools Version: 29.0.1
SDK Build Tools Version: 28.0.3

Build Information:
Mono: mono/mono/2019-02@e6f5369c2d2
Java.Interop: xamarin/java.interop/d16-2@d64ada5
LibZipSharp: grendello/LibZipSharp/d16-2@caa0c74
LibZip: nih-at/libzip/rel-1-5-1@b95cf3f
ProGuard: xamarin/proguard/master@905836d
SQLite: xamarin/sqlite/3.27.1@8212a2d
Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-2@6f6c969

=== Microsoft Mobile OpenJDK ===

Java SDK: /Users/bramin/Library/Developer/Xamarin/jdk/microsoft_dist_openjdk_1.8.0.25
1.8.0-25
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Android Device Manager ===

Version: 1.2.0.58
Hash: d03e3ea
Branch: remotes/origin/d16-2~1
Build date: 2019-07-16 17:37:14 UTC

=== Xamarin Inspector ===

Version: 1.4.3
Hash: db27525
Branch: 1.4-release
Build date: Mon, 09 Jul 2018 21:20:18 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 802001105
Git revision: a187f6852ae62ba52b944b47e53702957edfcf7b
Build date: 2019-07-22 18:20:10+00
Build branch: release-8.2
Xamarin extensions: 34bd2378d8d707df25989727c23ce68d9a232a67

=== Operating System ===

Mac OS X 10.14.6
Darwin 18.7.0 Darwin Kernel Version 18.7.0
Thu Jun 20 18:42:21 PDT 2019
root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64

=== Enabled user installed extensions ===

NuGet Package Management Extensions 0.18

brminnick picture brminnick on 25 Jul 2019
👍1

@briminnick - our current working theory is that this is an issue with mono. Please could you try this workaround and let me know if it resolves the issue?

owenniblock picture owenniblock on 25 Jul 2019
👍1

Thanks @owenniblock! Reverting to Mono v5.18.1.28 fixed it 馃憤

brminnick picture brminnick on 25 Jul 2019

Thanks for letting us know. I've filed an issue on the mono project here: https://github.com/mono/mono/issues/15830

owenniblock picture owenniblock on 25 Jul 2019
👍2

actually they've just closed it on Mono side with this message
UITest code needs to be updated to check for HttpRequestException.

cheles picture cheles on 1 Aug 2019
👍1

@cheles - I'm on it. My local dev set-up seems unhappy after I installed Xcode 11 so might be Monday before we get a dev release out.

owenniblock picture owenniblock on 1 Aug 2019
👍1

Please try the latest dev release of UITest and let us know if you hit further issues: https://www.nuget.org/packages/Xamarin.UITest/3.0.3-dev1

owenniblock picture owenniblock on 5 Aug 2019
🎉1 👍1

@owenniblock - I am seeing some improvement using 3.0.3-dev1 (I can now run the REPL on iOS) however I am still seeing the ThreadAbortException.

adamzucchi picture adamzucchi on 5 Aug 2019

Thanks @owenniblock!

The System.Net.Http.ConnectHelper.ConnectAsync exception I was experiencing is resolved with Xamarin.UITest v3.0.3-dev1; iOS tests now run as expected using Mono v6.0.0.311.

brminnick picture brminnick on 5 Aug 2019

@adamzucchi I'm not sure if this will resolve your error, but I saw this work-around in the Release Notes for Xamarin.UITest v3.0.3-dev1:

Known Limitations

[iOS] Suspected timing issue running on iOS locally may result in
connection failures to the test server running on a
simulator. Running Console.app and reading simulator logs while testing may offer a workaround for this issue.

brminnick picture brminnick on 5 Aug 2019

@brminnick - Thanks for the suggestion (which I had not previously seen) : )

Unfortunately nothing seems glaringly obvious when looking at the logs. I will report back if I come across anything that seems to have been the culprit.

Thanks again!

adamzucchi picture adamzucchi on 5 Aug 2019

@adamzucchi - I'm just looking at this with fresh eyes (I think it's been muddied slightly by additional issues we've seen lately).

The bit of code which is failing here is running a command like this:

xcrun xcodebuild test-without-building -xctestrun {xctestrunPath} -destination 'id={deviceId}' -derivedDataPath {derivedDataPath}

Do you have access to another development system you could test this on? If we can get a small project which replicates this consistently it would really help us work out what's going on here. If it works on another system, let's try to discount some Xcode weirdness (maybe removing and re-installing Xcode will make a difference).

owenniblock picture owenniblock on 5 Aug 2019

I no longer see any "Connection refused" in 3.0.3-dev1 . Thanks @owenniblock & @brminnick

cheles picture cheles on 7 Aug 2019

I am currently running my tests using NUnit 3.12, Xamarin.UITest 3.0.3
I no longer get the original error or the "connection refused" error

jpc1963 picture jpc1963 on 13 Aug 2019
👍4

Hang on. I don't think I tested this properly the last time.
I am (again) trying to update my UI test to nunit 3 using the following packages:

  • Nunit: 3.12
  • Specflow 3.0.225
  • Xamarin UITest 3.0.3

I am using VS2019 for mac (version 8.2.5 build 5). I have mono 6.0.0.327 but I have set 5.18.1.28 as the default (as this was proposed as a workaround). When I run my test (on ios simulator) I get the error: Exception has been thrown by the target of an invocation.

This is the reason I created the post originally. Have I missed some fix?

jpc1963 picture jpc1963 on 30 Aug 2019

Hi @jpc1963 - UITest 3.0.3 should be compatible with the latest mono.

Can I ask what version of Xcode you're using? And also whether you have Xcode 11 installed (regardless of whether you are using it for this simulator).

owenniblock picture owenniblock on 30 Aug 2019

I am using xcode 10.3. xcode 11 is not installed.

jpc1963 picture jpc1963 on 2 Sep 2019

One more update...
I removed all references to SpecFlow from my test project. And converted the specflow feature to a simple nunit test. This works OK using the latest version of nunit & xamarin.uitest.
I am running the tests from vs2019 on a mac. Using the latest mono version.
So the problem seems to be specflow - or the interaction between visual studio & specflow.

jpc1963 picture jpc1963 on 2 Sep 2019

Thanks @jpc1963 - let me record this over here and get someone to try and replicate. Is it possible to provide a sample project using SpecFlow which replicates this for our engineers?

owenniblock picture owenniblock on 4 Sep 2019

UploadTestApp.zip

@owenniblock here is a simple app & ui test.
Install the app onto an iphone simulator. Open the ui test solution, restore nugets and run the login test. You should get the error.

jpc1963 picture jpc1963 on 4 Sep 2019

Thanks - I'll add this to the ticket.

owenniblock picture owenniblock on 4 Sep 2019

Hi, @jpc1963

Thank you for taking the time to help diagnose this issue. I agree that it appears to be a problem with the interaction between Visual Studio and SpecFlow. However, we don't own Specflow and can't fix the issue. I would recommend taking a look at the SpecFlow bug tracker and see if there are similar issues there regarding Visual Studio as well as any potential workarounds.

Oddj0b picture Oddj0b on 5 Sep 2019

Another update..
I created an issue at specflow (here: https://github.com/techtalk/SpecFlow/issues/1726).
Their response is basically that since the logs show nothing relating to specflow itself, that the problem may lie with Xamarin.uitest.

jpc1963 picture jpc1963 on 18 Sep 2019

@jpc1963 Your link is set as this:
[https://github.com/techtalk/SpecFlow/issues/1726](url)
which points to https://github.com/microsoft/appcenter/issues/url

Corrected target: https://github.com/techtalk/SpecFlow/issues/1726

From that discussion:

No ideas on my side.
We don't maintain the SpecFlow VS Extension for VS for Mac/MonoDevelop/XamarinStudio and in the callstacks I don't see anything related to SpecFlow.
What I know is, that Xamarin UITest has some ugly hacks for SpecFlow support that is version dependent. Perhaps they didn't update it yet for SpecFlow 3.

I'm not sure the best way to resolve this "we say ____, they say _____" back and forth with SpecFlow, @Oddj0b or @owenniblock can we confirm if we still feel this is out of our scope and highlight data relevant to that?

King-of-Spades picture King-of-Spades on 18 Sep 2019

Just to clarify the quote above: The Mac integration for SpecFlow is not maintained by the SpecFlow team, but is a third-party solution. We only maintain the Windows version. So if the issue is in the SpecFlow Visual Studio extension for Mac, there's nothing the SpecFlow team can do about it. That would need to be addressed to the maintainer of the Mac Visual Studio extension project.

If the issue is in SpecFlow itself, then we can address it. I hope that makes it clear that Andi was just trying to clarify ownership and why he is unable to comment on the Mac extension.

Stephen-Mc picture Stephen-Mc on 19 Sep 2019
👍1

@Oddj0b is this problem still being looked at?
I know its closed but I dont think it has been resolved. I just downloaded the latest version of VS2019 for mac and I still see the problem

jpc1963 picture jpc1963 on 6 Nov 2019

Sorry, to comment on such an old issue but I just seem unable to get SpecFlow working with Xamarin.UITest to work. I have some working UITests that I've set up without SpecFlow but as soon as I add the SpecFlow and SpecFlow.NUnit packages all my tests fail with
System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ----> System.TypeLoadException : VTable setup of type Xamarin.UITest.SpecFlow.Shared.TestTracerDecorator failed
Is it not possible to combine 'vanilla flavour' uitesting with SpecFlow features, or am I missing something?

Jeanjean picture Jeanjean on 5 Mar 2021

So it turned out to be a compatibility issue between SpecFlow and Xamarin.UITest. Downgrading the SpecFlow nugets solved my issue. A working combination for me right now is:

  • NUnit 3.13.1
  • SpecFlow 3.5.14
  • SpecFlow.NUnit 3.5.14
  • Xamarin.UITest 3.0.15
Jeanjean picture Jeanjean on 5 Mar 2021
👍3

@Jeanjean Thank you so much for the info. I encountered the same issue, but with your help, I now have a working test pipeline, yeah!

Slm0nB picture Slm0nB on 6 Apr 2021

@Jeanjean What version of the Xamarin.UITest nuget are you using with NUnit 3.13.1 and SpecFlow 3.5.14 ?

barrymculhane picture barrymculhane on 6 Apr 2021

@Jeanjean What version of the Xamarin.UITest nuget are you using with NUnit 3.13.1 and SpecFlow 3.5.14 ?

@barrymculhane I'm currently on 3.0.15. I've edited my post to also include the Xamarin.UITest version.

Jeanjean picture Jeanjean on 7 Apr 2021

@Jeanjean This solved it for me. Thank you so much.

FYI, I also had to downgrade BoDi to version 1.4.1.

kristofberge picture kristofberge on 22 Apr 2021
🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

galfonso777 picture galfonso777  路  3Comments
vonovak picture vonovak  路  3Comments
Oddj0b picture Oddj0b  路  3Comments
busterlaursen picture busterlaursen  路  4Comments
sklmnv picture sklmnv  路  3Comments