Appcenter: Xamarin.UITest on iOS stopped restarting the app even with StartApp

[akamud]

What App Center service does this affect?
Xamarin.UITest

Describe the bug
Running iOS tests locally sometimes won't restart/reset my app, even when I call StartApp before each test. This makes my tests fail occasionally because the app is just opened in a different screen.

I don't recall this happening before (not sure when it started because I can't test older versions with XCode 10.2), so it looks like something that changed in recent versions.

To Reproduce
Steps to reproduce the behavior:

1. Clone this project: https://github.com/akamud/KickassUI.Traveler
2. Change the device UDID to your local iPhone X simulator on AppInitializer.cs
3. Run the tests once
4. Run the tests again

Expected behavior
The second test fails because the app is not restarted.

Note: this might happen on any run, but this was the most easy way I found to reproduce the problem.

Smartphone (please complete the following information):

• Device: iPhone X Simulator
• OS: 12.2
  
  
  
  • XCode: 10.2
  
  

Additional context
Xamarin.TestCloud.Agent: 0.21.8
Xamarin.UITest: 2.2.7.2002-dev

---

I'm seeing this behavior in all of my apps with UITests, even apps I used to run before with no problems.

[King-of-Spades]

Thanks for mentioning this, this is a known issue. We have a workaround by forcing the iOS simulator to restart between each test case:

Environment.SetEnvironmentVariable("UITEST_FORCE_IOS_SIM_RESTART", "1");

Here's an implementation by placing that variable just before your ConfigureApp.iOS method: https://github.com/King-of-Spades/AppCenter-Test-Samples/blob/master/Xamarin.UITest/UITestDemo/UITestDemo.UITest/AppInitializer.cs#L35-L36

[akamud]

Is the restart going to be the default in new versions or should I be using this workaround forever?

[King-of-Spades]

@Akamud we have an active bug report for the issue, but I don't have an ETA for the fix currently. When a fix is made it should be mentioned in the release notes for the Xamarin.UITest package, unless the fix is applied to the Test Cloud Agent instead (but I don't think the cause is directly in there.)

[akamud]

Awesome, thanks!

[Oddj0b]

Sorry to bring bad news, we still haven't made progress on this part. I will update when we know more.

[cheles]

Actually, restarting the simulator, will bring another issue when having multiple tests:

• in case the simulator doesn't start fast enough, the test will fail with
  SetUp : System.Net.Http.HttpRequestException : Connection refused
  ----> System.Net.Sockets.SocketException : Connection refused

Tried with our sample from here https://github.com/King-of-Spades/AppCenter-Samples/tree/master/Xamarin.UITest/FormsGallery

The only workaround I could find was to set UITEST_FORCE_IOS_SIM_RESTART to 0

[King-of-Spades]

Hmm...sounds like a bit of a catch-22. I don't consistently see that Connection refused error when using the SIM_RESTART method, but I do see it intermittently, so your observations are reasonably credible to me.

[cheles]

@King-of-Spades we have received some reports that this is happening constantly with these conf:
Xcode: 10.3 (10G8)
Nunit: 2.6.4
Xamarin.UiTest: 3.0.1
Visual Studio for MAC: 2019 (currently Up-To-Date, no available updates are suggested)
Mac OS: 10.14.6
Xamarin Forms Version we use for building the application: 4.1.0.618606
Mono JIT compiler version 6.0.0.311

Any idea why this might happen? This started after VS4m was updated to latest 8.2.

On my machine, with same config, I can only see it intermittently when moving between tests or when you have multiple tests.

[cheles]

Looks to be a regression in Mono 6.x https://github.com/mono/mono/issues/15830

[shoeIT]

Actually, restarting the simulator, will bring another issue when having multiple tests:

• in case the simulator doesn't start fast enough, the test will fail with
  SetUp : System.Net.Http.HttpRequestException : Connection refused
  ----> System.Net.Sockets.SocketException : Connection refused

Tried with our sample from here https://github.com/King-of-Spades/AppCenter-Samples/tree/master/Xamarin.UITest/FormsGallery

The only workaround I could find was to set UITEST_FORCE_IOS_SIM_RESTART to 0

oh thank god I'm not the only one, I could not find anything related to this problem. I got the connection refused all the time and cannot start the tests at all. Really frustrating since we are close to release...

[King-of-Spades]

@cheles Thanks for the data. I'll see if I can match my environment to yours and if that seems to influence how often this issue is occurring. With your observation of the Mono regression, did downgrading Mono seem to help with the symptoms?

[King-of-Spades]

I am seeing the Connection Refused error consistently now and am currently experimenting with workarounds.

[cheles]

as mentioned in above bug, downgrading to mono 5.x would be one workaround. the thing is latest VS 4 mac will complain every time if Mono is not 6.x.

[King-of-Spades]

This downgrade works for me. (Gathered from https://versions.internalx.com/products):

Visual Studio Community 2019 for Mac
Version 8.1.5 (build 9)
Installation UUID: f63ec00d-8c4e-415e-86b3-06ebd839d9ac
    GTK+ 2.24.23 (Raleigh theme)
    Xamarin.Mac 5.6.0.25 (d16-0 / 50f75273)
    Package version: 518010028

Mono Framework MDK
Runtime:
    Mono 5.18.1.28 (2018-08/223ea7ef92e) (64-bit)
    Package version: 518010028

Trying a few more experiments though.

[cheles]

8.1 works with 5.x but from 8.2 it will require every time to pick Mono 6.x and then downgrade. Net/Mono to 5.x

[King-of-Spades]

Yup, downgrading both is the only workaround I've been able to confirm. On the plus side, the downgraded Mono & VS4Mac do seem to still be playing nice with the latest versions of the UITest packages & Xcode 10.3.

I'm not sure if the Test team will be able to provide a patch for this behavior or if we'll have to wait for a Mono fix.

[King-of-Spades]

I think it's this issue: https://github.com/microsoft/appcenter/issues/421

It looks like a slightly cleaner solution is here: https://github.com/microsoft/appcenter/issues/421#issuecomment-515018570

Basically within VSMac Preferences, you can switch the version of Mono. I suspect it comes up because between major version changes it doesn't overwrite the previous major version. Also, there's an existing bug filed for the issue against Mono here: https://github.com/mono/mono/issues/15830

^ That should just about cover the problem @cheles encounters. I think, once those symptoms are worked around; the behavior in this original bug report is still valid.

[King-of-Spades]

Haha, the mono bug points back to UITest. Well, the silver lining is still that it looks like it's being worked on :D

[King-of-Spades]

Internal tracking id: ADO - 52641

[akamud]

Did this get any update?

[db2222]

Any new news? This still seems to be an issue which makes the UI tests for our company useless.
Thanks in advance!

[akamud]

Hi, is this being handled?

[King-of-Spades]

^ @Oddj0b any news?

[JohnHDev]

@King-of-Spades @Oddj0b is Xamarin.UITest deprecated? If not, why hasn't this been addressed? This issue was opened 2 years ago, and Im constantly hitting connection refused.
Issues like this waste hundreds of hours for developers. It would be better to just deprecate Xamarin.UITest if you can't make a stable version. Its one of the reasons why we are looking to move away from Xamarin entirely.

[King-of-Spades]

Hi @JohnHDev sorry for the frustration. I'm no longer on the App Center Test team, in fact I'm leaving Microsoft soon most likely. Xamarin.UITest did have a new version come out 5 days ago, though I don't see any explicit indicators that the issue has been updated: https://www.nuget.org/packages/Xamarin.UITest/

My best research, based on my limited remaining access (ending May 31st) indicates that the internal bug was canceled when the team (perhaps incorrectly) believed the reports on the issue dried up.

One of the nuances of these symptoms is that they can disappear or reappear with updates. The problem at a fundamental level is that many times in the past, Apple's Xcode updates (even minor ones) would break basic Xamarin.UITest functionality and require patching. To some extent, this could be a challenge for Google's Android updates, but not nearly as much to the same degree. (Other tooling dependencies can also have an impact, but Xcode / Android updates are the most common).

I suspect the best way to get the spotlight on your case is to do the following:

1. Fork / clone etc. this sample: https://github.com/microsoft/appcenter-Xamarin.UITest-Demo
2. Update to the latest packages (I used to maintain it, but as I said I'm not part of the team anymore.)
3. Modify the sample if needed to demo your issue / the failure state.
4. Log into App Center & Contact support with your version of the sample that reproduces the issue.

The reason this is effective is that it dramatically narrows the expected behavior & demonstrates reproducibility. If other devs like you are wanting to more collectively get attention on this you could always share the same "copy" of the sample to reproduce along with the tickets you open.

This strategy is also ideal, where feasible, for highlighting any other bugs you want to report.

Lastly, I'm really really sorry that it's so frustrating. I supported Xamarin Test Cloud / App Center Test for many years and always tried to advocate for fixing bugs, reducing issues that trip up developers, etc...It wouldn't be appropriate for me to say more about that, beyond my apology...but I wish you folks using the tooling all the best.

[JohnHDev]

@King-of-Spades I very much appreciate your candour, and I would like to apologise for my tone in my previous reply. I have no excuse other than the frustrating nature of some of the MS products can feel like banging my head against a wall.
I wish you the best in your future endeavours! Thank you for all your efforts with AppCenter and Xamarin.UITest.

As an aside I did get this working, and it is now working well. I don't know why it was failing for me, but I won't rule out user error on my part.

[King-of-Spades]

@JohnHDev No worries, 7 years in support also gave me a quite thick skin. :D

My own philosophy on user error is, it's really great when we can design software tools that follow the philosophy "falling into the pit of success" described in this article: https://blog.codinghorror.com/falling-into-the-pit-of-success/

It's not always possible to design tools in such a way as to reduce user confusion and/or user error...but oftentimes the pain and damage from those things can be reduced by being clearer about basic tool usage & debugging. That was actually part of the inspiration for creating the sample I shared, and a few other samples I've created over the years.

Actually, it's how I'm hoping to retool my career: I'm trying to find roles in doc writing and/or docs management full-time. Over that 7 years in support I've found that most engineers asking for help or reporting bugs have similar experiences to you: they just want the information to be unblocked and get back to building things. Most engineers don't even want "all the answers" given to them, but just the useful context that isn't easily discoverable from their perspective.

[JohnHDev]

@King-of-Spades that sounds like a good plan.

Actually, considering the internal bug for this has been closed, would you mind closing this too? Otherwise it leads to confusion where developers find an issue, google it, see an open bug with no recent movement and blame the tools, sometimes too quickly.

[JulesMoorhouse]

I've come up with what seems like a reliable workaround

        const string FullPathToIPA = "/Users/jm/repos/ourapp/ourapp/iOS/bin/iPhoneSimulator/Debug/ourapp.iOS.app";

                // Deleting and re-installing the app basically clears app data.
                // Delete app
                var processUninstall = Process.Start("xcrun", "simctl uninstall booted " + IpaBundleId);
                processUninstall.WaitForExit();

                // Install app
                var processInstall = Process.Start("xcrun", "simctl install booted " + FullPathToIPA);
                processInstall.WaitForExit();

                _app = ConfigureApp
                    .iOS
                    .DeviceIdentifier(SimIdiPhone8iOS124)
                    .PreferIdeSettings()
                    .AppBundle(PathToIPA)
                    //.InstalledApp(IpaBundleId)
                    .StartApp(AppDataMode.Clear); // Currently only supported in Xamarin Test Cloud.

I think the FullPathToIPA might be slightly different beyond the obvious with the additional iOS folder.

Also I'm betting the path can be derived from the current home folder etc.