Mono: Getting ObjectDisposedException related to MobileAuthenticatedStream

[bhaskar-infotmt]

We have a Xamarin mobile application that connects to an MQTT broker using https://github.com/eclipse/paho.mqtt.m2mqtt library.

We are getting ObjectDisposedException related to Mono.Net.Security.MobileAuthenticatedStream on Nokia 7 plus (HMD Global), Wileyfox and occasionally on Samsung Galaxy J7 (2016).

Cannot replicate but if happens on a device, it happens continuously for multiple runs (e.g. on a Connect() call, log below).

Application compiled with Xamarin.Android SDK version 9.1.4-2.

Related exception trace is below.

InnerException: uPLibrary.Networking.M2Mqtt.Exceptions.MqttCommunicationException ---> System.AggregateException: One or more errors occurred. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MobileAuthenticatedStream'.
at Mono.Net.Security.MobileAuthenticatedStream+d__58.MoveNext () [0x001bf] in <6fb28c247b244399bae1ab274efb33ed>:0
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <6eb4d101e3ff4f6aa2ed4600b42cd999>:0
at System.Threading.Tasks.Task1[TResult].GetResultCore (System.Boolean waitCompletionNotification) [0x0002b] in <6eb4d101e3ff4f6aa2ed4600b42cd999>:0 at System.Threading.Tasks.Task1[TResult].get_Result () [0x0000f] in <6eb4d101e3ff4f6aa2ed4600b42cd999>:0
at Mono.Net.Security.MobileAuthenticatedStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00018] in <6fb28c247b244399bae1ab274efb33ed>:0
at System.Net.Security.SslStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00006] in <6fb28c247b244399bae1ab274efb33ed>:0
at uPLibrary.Networking.M2Mqtt.MqttNetworkChannel.Receive (System.Byte[] buffer) [0x0000e] in :0
at uPLibrary.Networking.M2Mqtt.MqttClient.ReceiveThread () [0x0000d] in :0
--- End of inner exception stack trace ---
at uPLibrary.Networking.M2Mqtt.MqttClient.SendReceive (System.Byte[] msgBytes, System.Int32 timeout) [0x0008c] in :0
at uPLibrary.Networking.M2Mqtt.MqttClient.SendReceive (uPLibrary.Networking.M2Mqtt.Messages.MqttMsgBase msg, System.Int32 timeout) [0x0001b] in :0
at uPLibrary.Networking.M2Mqtt.MqttClient.SendReceive (uPLibrary.Networking.M2Mqtt.Messages.MqttMsgBase msg) [0x00000] in :0
at uPLibrary.Networking.M2Mqtt.MqttClient.Connect (System.String clientId, System.String username, System.String password, System.Boolean willRetain, System.Byte willQosLevel, System.Boolean willFlag, System.String willTopic, System.String willMessage, System.Boolean cleanSession, System.UInt16 keepAlivePeriod) [0x0005e] in :0

There are quite a number of issues related to MobileAuthenticatedStream on Xamarin & Mono forums, however we have not been able to come to resolution yet.

Any hint on this would be highly appreciated.

[david4neblio]

Same. I have a client running mono version 5.20 who is getting this task exception:

TaskScheduler.UnobservedTaskException: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.
Object name: 'MobileAuthenticatedStream'.) ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MobileAuthenticatedStream'.
  at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) [0x0024b] in <e8eb3d7a311640f484845e45cbec8973>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'MobileAuthenticatedStream'.
  at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) [0x0024b] in <e8eb3d7a311640f484845e45cbec8973>:0 <---

[david4neblio]

@baulig Please assist as this is mostly your code.

[marek-safar]

/cc @steveisok

[michaldobrodenka]

Most of my app crashes on Android are this crash. Using latest preview with 6.4.0 mono. I'm using websocket-sharp. Occurs probably when app goes foreground and try to reconnect websocket.

MobileAuthenticatedStream.StartOperation(Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) android.runtime.JavaProxyThrowable: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.caused.AggregateException(s) Object name: 'MobileAuthenticatedStream'.) ---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'MobileAuthenticatedStream'. Mono.Net.Security.MobileAuthenticatedStream.StartOperation(MobileAuthenticatedStream.OperationType type, AsyncProtocolRequest asyncRequest, CancellationToken cancellationToken)<f590fdd5bbfa4641a9851e2bc1fd80af>:0 --- End of inner exception stack trace --- ---> (Inner Exception #0) System.ObjectDisposedException: Cannot access a disposed object. Object name: 'MobileAuthenticatedStream'. Mono.Net.Security.MobileAuthenticatedStream.StartOperation(MobileAuthenticatedStream.OperationType type, AsyncProtocolRequest asyncRequest, CancellationToken cancellationToken)<f590fdd5bbfa4641a9851e2bc1fd80af>:0

[michaldobrodenka]

It seems that during await method in MobileAuthenticatedStream.StartOperation, instance of MobileAuthenticatedStream is falsely Garbage Collected and when execution returns to MobileAuthenticatedStream it tries to work with disposed object. A GC bug with async/await?

[cheles]

Hi @baulig , @steveisok . Are there any updates on this?

[clintonrocksmith]

Hello,

We are seeing this in our Crash reporting also... using latest iOS/Android/Mono and Visual Studio

Mono.Net.Security.MobileAuthenticatedStream.StartOperation(Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) in

[steveisok]

@baulig please dig into this one.

[bellissimo]

Same here. Problem seems to have got a lot worse on Android since a fairly recent version of VS (not sure exactly which, but since 16.3), with hundreds or crashes in the wild, though not been able to reproduce locally:

VS2019 16.3.4

[steveisok]

@baulig and I worked together on this one yesterday and I think we have an idea of what the problem may be. If our testing proves correct, I think we'll have a PR up soon.

We also think this is a mono problem in general and does not apply to one platform.

[clintonrocksmith]

Thank you guys, we appreciate your efforts

[SSELLIOTTHOME]

This issue has also happened to us, specifically on Android over a HTTPS connection whilst uploading large images. I can put in 'retry' code but obviously that is not an ideal solution. Cannot replicate in our local environment. Any thoughts for an ETA on this solution? like @clintonrocksmith I appreciate your efforts.

[SSELLIOTTHOME]

And for those of us still dealing with VS2017 and note made the upgrade yet - will there be a fix for the Xamarin.Android 9.1.x line?

[kspearrin]

I am also seeing this in production with our latest update to Xamarin Android 10 and VS 2019.

[clintonrocksmith]

Not sure if this is related, but we also are getting a lot of these....

• Unable to write data to the transport connection: The socket has been shut down.

System.Net.Sockets.NetworkStream.EndWrite(System.IAsyncResult asyncResult)

System.IO.Stream+<>c.b__58_1(System.IO.Stream stream, System.IAsyncResult asyncResult) in
System.Threading.Tasks.TaskFactory1+FromAsyncTrimPromise1[TResult,TInstance].Complete(TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) in

--- End of stack trace from previous location where exception was thrown ---
Mono.Net.Security.MobileAuthenticatedStream.InnerWrite(System.Boolean sync, System.Threading.CancellationToken cancellationToken) in
Mono.Net.Security.AsyncProtocolRequest.ProcessOperation(System.Threading.CancellationToken cancellationToken) in
Mono.Net.Security.AsyncProtocolRequest.StartOperation(System.Threading.CancellationToken cancellationToken) in
Mono.Net.Security.MobileAuthenticatedStream.StartOperation(Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) in

The other packages we have in our solution to do with networking include OkHttp for Glide, Refit and Polly

[clintonrocksmith]

@kspearrin and @SSELLIOTTHOME Do you have OkHttp, Refit or Polly in your solutions?

[kspearrin]

@clintonrocksmith I am using none of those.

[SSELLIOTTHOME]

I not using either of those libraries either. For me I think the issue is coming from a load balancer in the middle. Switching to http 1.0 gets around the problem.

[kspearrin]

@SSELLIOTTHOME

Switching to http 1.0 gets around the problem.

Switching to HTTP 1.0 on the client? How did you do this? Just change the Version on HttpRequestMessage?

[michaldobrodenka]

I'm getting this crash and I'm not using http - only websockets (websocket-sharp library)

[SSELLIOTTHOME]

@kspearrin Exactly that, just change the version on the HttpRequestMessage. I think Microsoft best practice is incorrect, mu suspicion is that if a connection is held open with a large upload, if it is a shared connection, in the time that the data is sent up the wire on a slow connection, load balancers and gateways that are less then generous will close the connection before it is fully sent. This is not normally a problem with downloading a web page so those IT depts will often use the 'its not our problem' line. I also found that putting in a simple delay before each request forces a new connection but that is far from an ideal solution

[kspearrin]

I changed all API calls to use requestMessage.Version = new Version(1, 0), but that doesn't seem to have helped. I am still seeing lots of these crashes reports in our app.

[SSELLIOTTHOME]

@kspearrin If you could check the http traffic using a proxy like Charles, you should see that the stay alive header is set to false, this would force it to create a new connection. Without seeing what the infrastructure is it would be difficult to see what is wrong. If you put in retry code, is the second attempt successful? - whilst this is a solution you need to put in code to prevent duplicate updates/inserts.

[clintonrocksmith]

This error is only showing up on Android for us. Not on iOS. I'll have a go at changing the values you've mentioned.

[clintonrocksmith]

We are still having this problem.
We have 5,000 instances of this in a month among 541 users

[ninaada]

Even we are facing this issue. Is there any workaround that works? We are seeing this in both VS for Mac 8.4 (buiild 2657) and in VS 16.4.2. Every build of our app in Android (iOS does not have this issue) keeps crashing and there is no way to handle this either.

[steveisok]

@ninaada When VS4M 8.5 becomes available, it'll include this fix. I'm pretty sure that'll solve your issue.

[mauruskuehne]

We have a similar error on iOS. We use Visual Studio for Mac 8.5 for building the app and Xamarin.iOS 13.16.0.13:

System.Threading.Tasks.TaskCanceledException: the operation was canceled. ---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'MobileAuthenticatedStream'. at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) ...

[StevenElecosoftShire]

I have never found a truly satisfactory solution to this problem other than an old fashioned retry (Polly is good for this). We also have code to cope with the same message being sent twice on the odd occasion that it happens. I find it disappointing that this is supposed to be acceptable on possibly the most critical and core part of any mobile application,

[DJSavage]

Also getting the same issue connecting to an Azure api. I am not keen on having to use Polly or similar as a workaround to fix a problem with a core part of any mobile application.

[StevenElecosoftShire]

I think think this only happens when there is a network problem - it is just that Xamarin does not handle it very well - perhaps the error looks worse than it really is. I wish MS would put the same time and effort they have into re-branding Xamarin Forms with that crazy name (as mentioned at BUILD) as they would in fixing this quite fundamental problem.