Xamarin-macios: System.ArgumentOutOfRangeException NsUrlSessionHandler

[czuvich]

Environment

Xamarin.iOS Application
iOS 12.3.1 and 11.4

Problem

I am getting the following exception in our iOS mobile Xamarin app.

Application Specific Information:
*** Terminating app due to uncaught exception 'System.ArgumentOutOfRangeException', reason: 'System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.'

Xamarin Exception Stack:
Parameter name: index
  at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) <0x100a1cf00 + 0x0004b> in <4db5e915761a454b810fa3913e31be34#17161b22946fba777b3431f3b3c47e37>:0 
  at Foundation.NSNotificationCenter.RemoveObserver (Foundation.NSObject observer) <0x1014ace20 + 0x00053> in <497181d9d26447828f8574819afeb3d5#17161b22946fba777b3431f3b3c47e37>:0 
  at System.Net.Http.NSUrlSessionHandler.RemoveInflightData (Foundation.NSUrlSessionTask task, System.Boolean cancel) <0x1015127dc + 0x0011f> in <0da8ced3e38d42039920e99e08ca9321#17161b22946fba777b3431f3b3c47e37>:0 
  at System.Net.Http.NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) <0x101512ec4 + 0x000bf> in <0da8ced3e38d42039920e99e08ca9321#17161b22946fba777b3431f3b3c47e37>:0

This issue has been recently introduced to our latest release which upgraded Xamarin.iOS. We see the issue when we are trying to make an HttpClient call within a UIApplication.BackgroundTask UIApplication.SharedApplication.BeginBackgroundTask. The exceptions seems to occur intermittently.

Steps to Reproduce

I can't quite reproduce the issue yet; however, I am getting crash logs via HockeyApp. I've also been able to identify the exact line of code in the Xamarin code that is throwing the exception. Line 191 in https://github.com/xamarin/xamarin-macios/blob/master/src/Foundation/NSUrlSessionHandler.cs.

Workaround

I'm probably just going to catch all exceptions during this web call.

Visual Studio Info

=== Visual Studio Community 2019 for Mac ===

Version 8.1 (build 2742)
Installation UUID: 5a3c7822-fd06-4d96-9af3-40f29763566e
    GTK+ 2.24.23 (Raleigh theme)
    Xamarin.Mac 5.6.0.25 (d16-0 / 50f75273)

    Package version: 518010028

=== Mono Framework MDK ===

Runtime:
    Mono 5.18.1.28 (2018-08/223ea7ef92e) (64-bit)
    Package version: 518010028

=== NuGet ===

Version: 5.0.2.5988

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
    2.1.11
    2.1.9
    2.1.8
    2.1.7
    2.1.2
    2.1.1
    2.0.5
    2.0.0
    1.1.1
    1.0.4
SDK: /usr/local/share/dotnet/sdk/2.1.700/Sdks
SDK Versions:
    2.1.700
    2.1.505
    2.1.504
    2.1.503
    2.1.302
    2.1.301
    2.1.4
    2.0.0
    1.0.3
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/5.18.1/lib/mono/msbuild/Current/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.6.10
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Updater ===

Version: 11

=== Apple Developer Tools ===

Xcode 10.2.1 (14490.122)
Build 10E1001

=== Xamarin.Mac ===

Version: 5.10.0.153 (Visual Studio Community)
Hash: 750a8798
Branch: d16-1-artifacts
Build date: 2019-04-30 15:17:54-0400

=== Xamarin.iOS ===

Version: 12.10.0.153 (Visual Studio Community)
Hash: 750a8798
Branch: d16-1-artifacts
Build date: 2019-04-30 15:17:54-0400

=== Xamarin.Android ===

Version: 9.3.0.22 (Visual Studio Community)
Commit: HEAD/8e7764fdf
Android SDK: /Users/Clay/Library/Developer/Xamarin/android-sdk-macosx
    Supported Android versions:
        5.0 (API level 21)
        6.0 (API level 23)
        7.0 (API level 24)
        7.1 (API level 25)
        8.0 (API level 26)
        8.1 (API level 27)

SDK Tools Version: 26.1.1
SDK Platform Tools Version: 28.0.1
SDK Build Tools Version: 28.0.1

Build Information: 
Mono: mono/mono/2018-08@3cb36842fc4
Java.Interop: xamarin/java.interop/d16-1@5ddc3e3
LibZipSharp: grendello/LibZipSharp/d16-1@44de300
LibZip: nih-at/libzip/rel-1-5-1@b95cf3f
ProGuard: xamarin/proguard/master@905836d
SQLite: xamarin/sqlite/3.27.1@8212a2d
Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-1@acabd26

=== Microsoft Mobile OpenJDK ===

Java SDK: /Users/Clay/Library/Developer/Xamarin/jdk/microsoft_dist_openjdk_1.8.0.25
1.8.0-25
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Android Device Manager ===

Version: 1.2.0.44
Hash: aac645b
Branch: remotes/origin/d16-1
Build date: 2019-05-29 19:55:24 UTC

=== Xamarin Designer ===

Version: 16.1.0.464
Hash: 66bb7b43f
Branch: remotes/origin/d16-1-new-document-model
Build date: 2019-06-07 07:10:32 UTC

=== Xamarin Inspector ===

Version: 1.4.3
Hash: db27525
Branch: 1.4-release
Build date: Mon, 09 Jul 2018 21:20:18 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 801002742
Git revision: ed27233de1b33083ea2234bc66a3b7824b99bbc7
Build date: 2019-06-12 10:18:51+00
Build branch: release-8.1
Xamarin extensions: 98dc4d7de2257c9b7342dee023910e16d02f8e65

=== Operating System ===

Mac OS X 10.14.5
Darwin 18.6.0 Darwin Kernel Version 18.6.0
    Thu Apr 25 23:16:27 PDT 2019
    root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64

[dalexsoto]

mmm interesting, it might be the RemoveInflightData and Dispose (GC) fighting over removing the notification, @mandel-macaque I think you worked on this, can you have a look?

[Hovejr]

Same issue, and appeared at the same time, never seen it before in an app 2 years old. Using Xamarin Forms version 3.4.0.1008975 and Enterprise version of VS for Mac, otherwise same information as over

[ashboul]

I am also receiving this problem frequently, it all started after getting the latest update!
I am unable to produce any steps

[czuvich]

This issue seems to have a larger impact than I thought as well. I am seeing the issue pop up on 3rd party libraries (i.e. HockeySDK.. see the stack trace below).

HockeySDK_iOSBindings_HockeyApp_iOS_BITHockeyManager_ConvertToNsExceptionAndAbort_object + 260

System.Collections.Generic.List<T>.get_Item(int index)
Foundation.NSNotificationCenter.RemoveObserver(NSObject observer)
System.Net.Http.NSUrlSessionHandler.RemoveInflightData(NSUrlSessionTask task, bool cancel)
System.Net.Http.NSUrlSessionHandler.NSUrlSessionHandlerDelegate.DidCompleteWithError(NSUrlSession session, NSUrlSessionTask task, NSError error)

[alanag13]

possibly related to #6443 ?

[czuvich]

So I tried catching the exception on the HttpClient async call, but the exception is still being propagated. Is there a workaround? If not, I think I’ll have to back out of the current Xamarin.iOS build, but I’m not sure which version broke it.

[dirar99]

I use Xamarin.Forms and I'm getting the same issue. Not sure what options I have to get around this.

[Phenek]

Yep it's annoying! It occur at each startup..
Hope someone will find a workaround.

[OmegaPrimeZ3]

Same issue, any update on when this can be addressed?

[mandel-macaque]

This looks to be an issue with the NSNotificationCenter, rather than the NSUrlSessionHandler, but it is exposed when we try to remove the observer when there are not request inflight to minimise the extra work done. Will take a look at what is going on with the notification centre.

[mandel-macaque]

@dalexsoto Looks like we might have an when trying to remove the observer. In order to understand, we have the following two methods (one generated, the other done manually):

// GENERATED CODE
public virtual void RemoveObserver (NSObject observer)
{
    if (observer == null)
         throw new ArgumentNullException ("observer");
    if (IsDirectBinding) {
        global::ObjCRuntime.Messaging.void_objc_msgSend_IntPtr (this.Handle, Selector.GetHandle ("removeObserver:"), observer.Handle);
    } else {
         global::ObjCRuntime.Messaging.void_objc_msgSendSuper_IntPtr (this.SuperHandle, Selector.GetHandle ("removeObserver:"), observer.Handle);
    }
    RemoveObserversFromList (observer, null, null);
}

// MANUAL CODE
void RemoveObserversFromList (NSObject observer, string aName, NSObject anObject)
{
    lock (__mt_ObserverList_var) {
        for (int i = __mt_ObserverList_var.Count - 1; i >= 0; i--) {
            ObservedData od = __mt_ObserverList_var [i];

            if (observer != od.Observer)
                continue;

            if (aName != null && aName != od.Name)
                continue;

            if (anObject != null && anObject != od.Object)
                continue;

            __mt_ObserverList_var.RemoveAt (i);
        }
    }
}

Perse, with the above code, the handler should not be worried about removing the observer more than once, since the notification centre should be thread safe. So we might be looking at an issue BEFORE this commit landed: https://github.com/xamarin/xamarin-macios/commit/dda50a78c41547961f5033436ec9ab127d0cc105

I think that the safest bet is to lock in the removal of the observer object in the handler. Ensuring that way that if the removal is called twice, we fully complete the execution making the check in https://github.com/xamarin/xamarin-macios/blob/master/src/Foundation/NSUrlSessionHandler.cs#L190 work correctly.

[mandel-macaque]

@czuvich the issue is already fixed in master (about 3 months ago). I'll do a backport to ensure that the fix is present ASAP.

[Phenek]

Thanks

[mandel-macaque]

@Phenek @czuvich if you want, you can try the preview channel, which should already have the fix.

[czuvich]

@mandel-macaque Thanks for the update! We generally don't use Preview updates in our app (I've hacked in a fix for now). Hopefully, the fix will be released into production soon!

[ghost]

We have tried VS Mac 2019 v8.2 Preview. So far the crash no longer occurs.

With today's v8.1.5.9, the issue can repro easily.

[mandel-macaque]

@clementyau thanks for the confirmation :)

I don't think we will have a SR for d16-1 so @czuvich could share his workaround until people can update.

[czuvich]

The workaround for our app is rather specific. The app was crashing on startup since we were performing a PushNotification registration AND a web sync both to our backend server in parallel. Rather than performing both at the same time, I wait until the sync is finished before invoking the PushNotification registration. Someone earlier posted a workaround that involved serializing HTTP calls (which is where I got the idea).

Having said that, the exception will probably still occur, but hopefully not nearly as often. Having said that, @mandel-macaque I really hope this issue is resolved soon. I'm guessing not everyone is going to be able to serialize all HTTP calls that use HttpClient. I understand development cycles take some time; however, I think HttpClient is effectively broken for all users on this version right?

[mandel-macaque]

@czuvich The simples workaround atm is to use a background session, that will require no code changes in the apps and will ensure that the issue does not happen.

[tipa]

@czuvich The simples workaround atm is to use a background session, that will require no code changes in the apps and will ensure that the issue does not happen.

Unfortunately this workaround isn't possible when using a library that uses NsUrlSessionHandler.
Would it be possible to re-open this issue until the fix is available?

[jmblargent]

Any update on which stable release this fix is going to be included on?

[akamud]

From reading the related issues, I'm really confused whether this is fixed or not, and what this has to do with Visual Studio, since it looks like a _macios_ issue. This PR #6537 was never merged, how can it be fixed?
Can anyone clarify please? /cc @spouliot @rolfbjarne

[rolfbjarne]

@akamud The PR #6537 was never merged because it wasn't the right fix, and this problem is fixed because another (previous) change was the real fix (https://github.com/xamarin/xamarin-macios/commit/dda50a78c41547961f5033436ec9ab127d0cc105).

However, there seems to be additional problems in this area (#6704, #6691), but unfortunately we never had a test case (and still don't), which makes it impossible to verify that a particular fix doesn't just uncover another problem.

If you still have problems with NSUrlSessionHandler, please file a new issue and attach all the information you have about the exception/crash.

[rolfbjarne]

@jmblargent the current stable release has this fix.

[Phenek]

From AppCenter:
NSNotificationCenter.RemoveObserver (Foundation.NSObject observer)
/Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.110/src/Xamarin.iOS/Foundation/NSNotificationCenter.g.cs, line 178

SIGABRT: Value cannot be null. Parameter name: observer

NSNotificationCenter.RemoveObserver (Foundation.NSObject observer) /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.110/src/Xamarin.iOS/Foundation/NSNotificationCenter.g.cs:178
NSUrlSessionHandler.RemoveNotification () /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.110/src/Xamarin.iOS/Foundation/NSUrlSessionHandler.cs:191
NSUrlSessionHandler.RemoveInflightData (Foundation.NSUrlSessionTask task, System.Boolean cancel) /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.110/src/Xamarin.iOS/Foundation/NSUrlSessionHandler.cs:220
NSUrlSessionHandler+NSUrlSessionHandlerDelegate.DidCompleteWithError (Foundation.NSUrlSession session, Foundation.NSUrlSessionTask task, Foundation.NSError error) /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.110/src/Xamarin.iOS/Foundation/NSUrlSessionHandler.cs:668

[rolfbjarne]

@Phenek that looks like #6691. Please add any information you have to that issue (the complete crash report at least)