Aspnetcore: Run automated testing in Xamarin scenarios

[JamesNK]

The SignalR C# client unit tests should be run on Xamarin iOS and Android to detect bugs that are specific to those runtimes.

Example of a C# client bug that is specific to Android: https://github.com/aspnet/SignalR/issues/2232

[davidfowl]

Yea, I'm not sure that's worth the overhead. I'd wait to see how many OS specific bugs come in before we start bootstrapping infrastructure for other environments.

[anurse]

This could fit in to a category of testing that we do on a schedule rather than per-commit

[anurse]

I'm updating this to be a little more general. I don't see a way to automate running unit tests in something like VS App Center, all the testing there is around apps. I think the right approach might be to have a test app with some simple automated testing for core scenarios.

[JamesNK]

I know UWP testing is basically a harness app that executes unit tests when it runs. There must be a Xamarin equivalent. We should reach out to MS Xamarin contacts for advice on unit testing a library.

[JamesNK]

good news: xunit appears supported in Xamarin https://gregshackles.com/testing-xamarin-apps-getting-started-with-xunit/

[JamesNK]

Lots of results: https://www.google.com/search?q=xunit+xamarin

Might still be worth reaching out to them for automation. Surely other teams that build client SDKs, e.g. Azure, are doing some automated testing on Xamarin.

[anurse]

Interesting. My concern is that App Center doesn't seem to support those natively, but maybe we could do something hacky like we do with Selenium and have App Center automate the Test Runner apps. I want to try and avoid having to have our own devices :).

Maybe we don't need devices though, maybe testing on the simulators is sufficient.

[JamesNK]

I believe Xamarin has hosted test environments as a service.

[JamesNK]

Yes it does: https://www.xamarin.com/test-cloud

[anurse]

Yeah, but UI tests, not Unit Tests: https://docs.microsoft.com/en-us/appcenter/test-cloud/supported-frameworks.

As I said, we can make that work, but I don't think it natively supports Unit Tests (and all the reporting we'd want with that)

[freever]

I've used this before to run tests on iOS, Android and UWP devices:
https://github.com/nunit/nunit.xamarin

But I never managed to get it playing nicely with UiTest so I couldn't get them running in an automated fashion in Test Cloud.

[anurse]

This is blocked on #1886

[anurse]

Rolling this up in to #11821