Aspnetcore: .NET Runtime Exception: ObjectDisposedException

[sepehr1014]

Describe the bug

After upgrading to ASP.NET Core 2.2, we frequently face this error and our website gets restarted. This happens once an hour or so. We had to target 2.1 again to fix this issue.
Here are the errors from event viewer (in the order they have occurred):

Application: dotnet.exe
CoreCLR Version: 4.6.27110.4
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.IO.FileStream.FileStreamCompletionSource.CompleteCallback(UInt64 packedResult)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Faulting application name: dotnet.exe, version: 2.2.27110.6, time stamp: 0x5be769ed
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2457, time stamp: 0x5b7e2adb
Exception code: 0xe0434352
Fault offset: 0x0000000000033c58
Faulting process id: 0x34374
Faulting application start time: 0x01d48cab8dc6d688
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Faulting package full name: 
Faulting package-relative application ID: 

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_ade641d7

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 4
Hashed bucket: 

Fault bucket 1396378374383491433, type 5
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: C:\Program Files\dotnet\dotnet.exe
P2: 2.2.27110.6
P3: 5be769ed
P4: System.Private.CoreLib
P5: 4.6.27110.4
P6: 5be75665
P7: 27fd
P8: b
P9: System.ObjectDisposedException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_C__Program Files_55e13f92ac2eada5e8dd2daa7dea78fd1e8d5d_0e1906bf_69a648dc

Analysis symbol: 
Rechecking for solution: 0
Report Id: aaeaddc7-02d9-45ce-9f29-318127ca0951
Report Status: 0
Hashed bucket: 0fdfa957dcca13d25360eebbb12a5969

To Reproduce

Steps to reproduce the behavior:
Use SDK v2.2.100 with IIS on Windows Server 2016

Additional context

Add any other context about the problem here.
Include the output of dotnet --info

Host (useful for support):
  Version: 2.2.0
  Commit:  1249f08fed

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[Tratcher]

Do you have any idea what application code or event triggered this?

This may need to be moved to https://github.com/dotnet/corefx/issues.

[sepehr1014]

@Tratcher Unfortunately not. I was hoping the stack trace would shed some light for you guys. Is it related to files and streams? We're using static files module but there are few actions that return 'File'. We also make extended use of web sockets.

[RoySalisbury]

I am also getting this. Just start the app and refresh a web page a few times and get the

System.ObjectDisposedException: 'The CancellationTokenSource has been disposed.'

Nothing special about my web app as far as I can tell. I thought it was my separate hosted service, but I took that out and it still has the issue.

[RoySalisbury]

Pretty sure this has nothing to do with it, but I removed

app.UseStaticFiles()

And I no longer have the issue. Could just be taking longer now, -or- it has something to do with the loading of css/js files....

[RoySalisbury]

OK.. Left in the 'UseStaticFiles' call but removed every js/css file being loaded. Still happened. But removed the 'UseStaticFiles' and after 10 min of constant refresh, no error. Usually it will happen within 4 or 5 page refreshes.

So, now, what is the call to 'UseStaticFiles' doing that either causes it, or makes it happen?

[Tratcher]

StaticFiles passes the HttpContext.RequestAborted CancellationToken to the read and write operations when it's serving a file. When you repeatedly press F5 some operations do not complete and get aborted. That's standard stuff and nothing has changed there in a long time.

What might be new in 2.2: IIS applications are hosted in-process by default (as opposed to the prior out-of-process mode). It may be disposing of the RequestAborted CTS after canceling it. @jkotalik an aborted CTS should not be disposed, it causes races like this.

Can you reproduce this if you edit the csproj and change it from InProcess to OutOfProcess? Or if you run only with Kestrel?

Reguardless, there's a bug in CoreFx FileStreamCompletionSource about throwing exceptions on background threads that we should file.

[jkotalik]

@Tratcher I think @sepehr1014 issue isn't related to inproc as the faulting exe is dotnet.exe.

I looked at how we handle request aborted in InProc and from what I can tell, we don't call dispose on the cts after cancelling it. We can add some debug asserts to confirm.

[RoySalisbury]

I'm actually running with Kestrel. And even waiting between refreshes of the page still causes the issue. So I don't think its the 'F5' aborting things.

[Tratcher]

I did find what looks like a regression in FileStreamCompletionSource (need to confirm which versions), but that doesn't explain why the CTS was disposed.

[Tratcher]

Hmm, the FileStreamCompletionSource happened in 2.1, not 2.2. There may be an additional change in Kestrel's CTS handling in 2.2...

[RoySalisbury]

Looking at the changes linked above, the only thing that jumped out at me was the following:

 // Free up the native resource and cancellation registration
 CancellationToken cancellationToken = _cancellationRegistration.Token; // access before disposing registration
 ReleaseNativeResource();

And that's only because in ReleaseNativeResource() it calls cancellationRegistration.Dispose() and I am not sure what that call does to the internals of the original Token reference. Could it be doing something to the original token?

I think I am just going in circles now. I traced back some discussions that may or my not be relevant..

https://github.com/dotnet/corefx/issues/14903

There was a mention in that thread about how Kestral does something different to handle an issue. Then once I dug into that I found this nugget:

           public void TryDispose()
           {
                // This normally waits until the callback is finished invoked but we don't care
                if (_callbackWrapper.TrySetInvoked())
                {
                    // Bug #1549, .NET 4.0 has a bug where this throws if the CTS
                    _registration.Dispose();
                }
            }

So your guess that Kestrel does something different is valid. They have an entire extension class just for CancellationTokens.

I switched over to IISExpress on my dev box and the issue seems to have gone away (another indication that its Kestrel). My end product is AM32 (Pi) which will be Kestrel, so I will do some testing there to see if its a platform specific thing.

However, MY issues seems that it has diverged from the original issue because he WAS getting it to happen in IIS and not Kestrel. Could be that the UseStaticFiles on Kestrel just made it show up quicker and something else in IIS made it happen (but for the same underlying cause).

[sepehr1014]

Our app is behind IIS and we're not using InProc mode. (because of https://github.com/dotnet/core/blob/master/release-notes/2.2/2.2-known-issues.md)
And I can confirm this bug is NOT present when targeting 2.1.6.

[abergs]

I'm running on Azure AppServices. I'm getting this issue when running InProc.

Here's output from Kudo (although I tried to run 2.2.103 with self contained, no difference) :

Kudu Remote Execution Console
Type 'exit' then hit 'enter' to get a new powershell process.
Type 'cls' to clear the console

PS D:\home> dotnet --version
dotnet --version
2.2.102

PS D:\home> dotnet --info
dotnet --info
.NET Core SDK (reflecting any global.json):

 Version:   2.2.102

 Commit:    96ff75a873



Runtime Environment:

 OS Name:     Windows

 OS Version:  10.0.14393

 OS Platform: Windows

 RID:         win10-x86

 Base Path:   D:\Program Files (x86)\dotnet\sdk\2.2.102\



Host (useful for support):

  Version: 2.2.1

  Commit:  878dd11e62



.NET Core SDKs installed:

  1.1.10 [D:\Program Files (x86)\dotnet\sdk]

  2.1.500 [D:\Program Files (x86)\dotnet\sdk]

  2.1.503 [D:\Program Files (x86)\dotnet\sdk]

  2.2.102 [D:\Program Files (x86)\dotnet\sdk]



.NET Core runtimes installed:

  Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.All 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.All 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]

  Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.AspNetCore.App 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.AspNetCore.App 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]

  Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

  Microsoft.NETCore.App 2.2.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]



To install additional .NET Core runtimes or SDKs:

  https://aka.ms/dotnet-download

PS D:\home> 

My exception is as follows:

System.ObjectDisposedException at System.Threading.CancellationTokenSource.ThrowObjectDisposedException
System.Threading.CancellationTokenSource.ThrowObjectDisposedException line:0
Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<>c__DisplayClass314_0.<AbortIO>b__0 line:0

[Doomblaster]

I'm experiencing the same issue, at least I'm quite sure it's related. I'm not able to reproduce it locally though, but no matter what I update of external nuget packages with these partial fixes, it seems to fail. I'm targeting full framework, net471. Currently running with Kestrel in service fabric.

PackageReferences:

<PackageReference Include="Microsoft.AspNetCore" Version="2.2.0" />
<PackageReference Include="Microsoft.AspNetCore.Http" Version="2.2.2" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.2.0" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Core" Version="2.2.2" />
<PackageReference Include="Microsoft.AspNetCore.Routing" Version="2.2.2" />
<PackageReference Include="Microsoft.AspNetCore.Server.IIS" Version="2.2.2" />
<PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="2.2.1" />
<PackageReference Include="Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets" Version="2.2.1" />
<PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.2.0" />
<PackageReference Include="Microsoft.AspNetCore.WebSockets" Version="2.2.1" />
<PackageReference Include="Microsoft.ServiceFabric.AspNetCore.Kestrel" Version="3.3.638" />

Normally I'd just reference AspNetCore.Mvc and AspNetCore, but in an effort to try to fix this I added updates direct references wherever I could find a newer one.

StackTrace:
System.ObjectDisposedException: The CancellationTokenSource has been disposed. at System.Threading.CancellationTokenSource.ThrowObjectDisposedException() at System.Threading.CancellationTokenSource.ThrowIfDisposed() at System.Threading.CancellationTokenSource.InternalRegister(Action`1 callback, Object stateForCallback, SynchronizationContext targetSyncContext, ExecutionContext executionContext) at System.Threading.CancellationToken.Register(Action`1 callback, Object state, Boolean useSynchronizationContext, Boolean useExecutionContext) at System.Threading.CancellationToken.Register(Action`1 callback, Object state) at System.IO.Pipelines.PipeAwaitable.AttachToken(CancellationToken cancellationToken, Action`1 callback, Object state) at System.IO.Pipelines.Pipe.FlushAsync(CancellationToken cancellationToken) at System.IO.Pipelines.Pipe.DefaultPipeWriter.FlushAsync(CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TimingPipeFlusher.FlushAsync(MinDataRate minRate, Int64 count, IHttpOutputAborter outputAborter, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.WriteAsync(ReadOnlySpan`1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.WriteDataAsync(ReadOnlySpan`1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(ReadOnlyMemory`1 data, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at Microsoft.AspNetCore.Http.Extensions.StreamCopyOperation.<CopyToAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.AspNetCore.StaticFiles.StaticFileContext.<SendAsync>d__49.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.<Invoke>d__7.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)

I would assume that this possibly should have been fixed by https://github.com/aspnet/AspNetCore/pull/4447, but...I'm struggling with figuring out if this even has been released and which nuget package needs to be updated in that case.

[Tratcher]

4447 was reverted, the fix was incomplete. @halter73

[Doomblaster]

Well, i still find it weird that there is no v2.2.1 or v 2.2.2 of Microsoft.AspNetCore.Server.Kestrel.Core since the code at least has namespace aligned with that. Anyways, let's see if @halter73 can share some insight. Should at least be part of 2.2.2 release as far as I understand https://github.com/aspnet/AspNetCore/issues?utf8=%E2%9C%93&q=milestone%3A2.2.2+label%3Aservicing-approved

[jkotalik]

To clarify, within the 2.2.2 release, #4447 was added and reverted. It was never released at all. Kestrel.Core hasn’t been patched in 2.2.x yet.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Espen Berglund [e-mail]
Sent: Tuesday, February 19, 2019 10:09:53 AM
To: aspnet/AspNetCore
Cc: Justin Kotalik; Mention
Subject: Re: [aspnet/AspNetCore] .NET Runtime Exception: ObjectDisposedException (#4422)

Well, i still find it weird that there is no v2.2.1 or v 2.2.2 of Microsoft.AspNetCore.Server.Kestrel.Core since the code at least has namespace aligned with that. Anyways, let's see if @halter73https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhalter73&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7C433c045aec2e4a6b95c608d69684af3d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636861893956778651&sdata=ZBcBacv6GLdtc%2BXhijvZp%2FpHc1YTTr4IbrtfCxG7OVQ%3D&reserved=0 can share some insight.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faspnet%2FAspNetCore%2Fissues%2F4422%23issuecomment-465195967&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7C433c045aec2e4a6b95c608d69684af3d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636861893956788659&sdata=MiUwrhHIYnIc9EcbDIQJJiK2FOsFdETgGoBojCgsDOw%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAH6uFbrlRu3huS8vk5MRUD8lBzo7ZVoeks5vPCHRgaJpZM4ZDA3p&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7C433c045aec2e4a6b95c608d69684af3d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636861893956798676&sdata=8idVFuD7EcVYUnoIH9CAd1gZXeGDiAyX33Cg8um4qDU%3D&reserved=0.

[halter73]

Ultimately, only a fix for referencing CancellationTokenRegistration.Token after the CancellationTokenSource has been disposed made the patch. This prevents the process crash. We will need to rework #4447 to not potentially wait for user code under an internal lock before it can released.

[muratg]

Triage: we think this is on Kestrel, but we need to validate what IIS is doing.

[HassanHashemi]

@Tratcher We are experiencing same issue, After requesting many files (some of the requests are killed before being completed the issue happens).
Pure Kestrel.

Reguardless, there's a bug in CoreFx FileStreamCompletionSource about throwing exceptions on background threads that we should file.

However i am 99% percent sure this is the case.

[Tratcher]

CoreFx patched this in 2.2.1 https://github.com/dotnet/coreclr/pull/21417

[vytautask]

I'm completely lost now. @Tratcher says it's patched and @jkotalik says it was was reverted and was never released. So, which one is it? Both? :)
My experience is that with 2.2.1 it's happening (currently on our production servers) kinda randomly (seems like not during the request but rather later on):

[2019-03-13 14:15:00.20] [<server_name_here>]   [Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer] [ERR] Connection ID ""17870283413211081584"", Request ID "null": An unhandled exception was thrown by the application.

System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.Threading.CancellationTokenSource.Cancel()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.<>c__DisplayClass314_0.<AbortIO>b__0(Object t)

P. S. We're using HttpClientFactory extensively throughout our application and initializing quite a bunch for requests with the original HttpContext cancellation token for those.
This was not the case before migrating to aspnet core 2.2.1 and InProcess (this exception was not happening).

[halter73]

@vytautask The bug you're observing (which is that the CancellationTokenSource backing the RequestAborted token can be disposed prior to middleware completing) still exists.

@Tratcher is referring specifically to a fix for referencing CancellationTokenRegistration.Token after the CancellationTokenSource has been disposed. This prevents the process crash and made it in to 2.2.1. We will need to rework #4447 for the full fix.

[jkotalik]

@halter73 to note, we now know this bug exists with ANCM InProcess.

[mqudsi]

We are also seeing this in production only, running under IIS with the new in-proc option. I have the logs enriched with the request context (Serilog integration) and it seems the error is never associated with a request, or at least the details of the request context are never present when the exception occurs. We don't have post-delivery processing of async procedures queued up after the client payload is delivered, does that mean it is safe to assume this isn't impacting any connected clients, ie it is not causing any external requests to prematurely fail?

[HassanHashemi]

@mqudsi Upgrading to 2.2.2 fixed the issue for me.

[halter73]

We don't have post-delivery processing of async procedures queued up after the client payload is delivered, does that mean it is safe to assume this isn't impacting any connected clients, ie it is not causing any external requests to prematurely fail?

Correct. The issue with prematurely disposing the CancellationTokenSource backing the RequestAborted token only occurs if the client already closed the underlying connection during request processing.

[jmk92]

I am very troubled by this now, even if we upgrade to the latest 2.2.3, or 3.0.0 preview 3, as long as hosting in the process, it will be irregular, has been a random problem.

We even upgraded from Windows Server 2016 to 2019, but the problem remains.

The log records are as follows:

2019-04-07 08:11:51.7792 | 2 | ERROR | Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer | Connection ID "154203257202027563", Request ID "8000002c-0005-d600-b63f-84710c7967bb": An unhandled exception was thrown by the application. Disposed Exception: The Cancellation has been disposed of Token Source.

At System. Threading. Cancellation TokenSource. ThrowObjectDisposedException ()

At System. Threading. Cancellation TokenSource. Cancel ()

At Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext. <>c_DisplayClass314_0. <AbortIO>b_0(Object t)

However, we have a machine equipped with SSMS, every time the log record above appears, the error window of VS will pop up. We can only prohibit pop-up through the registry.

Fortunately, we switched to Kestrel and there was no problem, but you know that in IIS hosting, you sacrificed 300% performance improvement, and I don't know which way to go now.

[jkotalik]

@jmk92 Did you hit this issue at all using Kestrel? From our initial investigation, we think this issue existed in both servers, but it could be more likely to happen with IIS (and after reading some code, I think it is significantly more likely).

[jmk92]

@jkotalik We have been using IIS, maybe both servers have this problem, but when we use InProcess, the error log has not stopped at all, several or even dozens of the same error messages a day.

2019-04-07 08:11:51.7792 | 2 | ERROR | Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer | Connection ID "154203257202027563", Request ID "8000002c-0005-d600-b63f-84710c7967bb": An unhandled exception was thrown by the application. Disposed Exception: The Cancellation has been disposed of Token Source.
At System. Threading. Cancellation TokenSource. ThrowObjectDisposedException ()
At System. Threading. Cancellation TokenSource. Cancel ()
At Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext. <>c_DisplayClass314_0. b_0(Object t)

However, when using outprocess (Kestrel), no similar error log has occurred, which may not mean that there is no problem.

When we use InProcess, we also have problems with #6834 , #9278 , which makes Kestrel feel more stable, at least for now.

We still hope to use InProcess. Maybe once you solve this problem, other chain reaction problems will be solved together.

[abergs]

Glad to see this has been worked on by @jkotalik. Any ballpark guess regarding which version this will be shipped in? (Patch for .net core 2.2.x or .net core 3?)

[jmk92]

@abergs I think, in the current situation, it may be updated to Preview 5.

Preview 4 will be released soon, and Preview 5 will take more than a month, or even two to three months.

2.2.4 just released a few days ago, 2.2.5 will take at least one month, so it seems.

We have to wait at least one month, no matter which one...

[Tratcher]

Right, preview5 is the next regular 3.0 release. We're also considering this for a patch in 2.2.6 or later.

[filipw]

FWIW I have encountered this issue:

{
  "StackTraceString": "   at void System.Threading.CancellationTokenSource.ThrowObjectDisposedException()\r\n   at void System.Threading.CancellationTokenSource.Cancel()\r\n   at bool Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.AbortIO()+(object t) => { }",
  "Message": "The CancellationTokenSource has been disposed.",
  "HResult": -2146232798,
  "RemoteStackTraceString": "",
  "ClassName": "",
  "RemoteStackIndex": -1,
  "Depth": 0,
  "Source": "System.Private.CoreLib",
  "HelpURL": null
}

8 times in the past week, from a total traffic of roughly 3 million requests in that timeframe.

This is on Azure App Service, .NET Core 2.2.4 and using ANCM InProcess.

[abergs]

@Tratcher I couldn't see this issue mentioned in the 2.2.6 release notes. Did I miss it or will it not be released for 2.x?

[Tratcher]

@jkotalik did we patch IIS for this?

[jkotalik]

No, we didn't. I don't believe we patched Kestrel either though?

[jeremycook]

I've seen the following exception 80 times in the last 7 days on a site that gets about 20K page views per week. Windows 2012 R2, dotnet hosting 2.2.6, InProcess.

SourceContext
Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer

Exception
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
   at System.Threading.CancellationTokenSource.ThrowObjectDisposedException()
   at System.Threading.CancellationTokenSource.Cancel()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.<>c__DisplayClass314_0.<AbortIO>b__0(Object t)

[rgelb]

I see that this issue was patched for 3.0. Will there be a patch 2.x line?

[Tratcher]

@shirhatti

[gaborbuzasi]

I would be very curious if there are any updates on this one related to .NET Core 2.2.x users? @Tratcher @shirhatti

[shirhatti]

There are two issues tracked in this thread.

1. CancellationTokenRegistration.Token throws an ODE if the associated CancellationTokenSource has been disposed. This was fixed in CoreCLR. This should be sufficient to prevent any process crashes.

2. There remains an issue where this a race between where the requestAborted token may be valid when you register it, but may have been disposed prior to using (due to a client disconnect). This results in an ODE which we catch and log as an application error, but it is benign.

We've re-worked the code to remove the chance of throwing an ODE in 3.0, however this doesn't meet the bar for a 2.2 backport since it's benign. We'd be happy to look into this again for 2.2 if anyone is actually experiencing a process crash due to this.

[SteveKennedy]

@shirhatti I don't understand. For the first issue, when you say "This was fixed in CoreCLR.", what can I do to obtain that fix for my applications? When is that fix available for .NET Core 2.2 apps?

[gaborbuzasi]

@shirhatti The problem is that you may have solved that the process doesn't crash, but when this error happens the response from the API isn't returned and it clears out all the CORS headers too. So users of our API need to keep refreshing their application until the error doesn't come up.
I don't think upgrading to a preview version of a framework (3.0) is a solution to this.

[Tratcher]

I don't think upgrading to a preview version of a framework (3.0) is a solution to this.

Why not? 3.0 is almost done, the latest previews have go-live (production) support licences. The final release will be out by the end of the month.

[halter73]

The problem is that you may have solved that the process doesn't crash, but when this error happens the response from the API isn't returned and it clears out all the CORS headers too. So users of our API need to keep refreshing their application until the error doesn't come up.
I don't think upgrading to a preview version of a framework (3.0) is a solution to this.

Even before the fix in 3.0, the underlying connection must have already been aborted for the CancellationTokenSource backing HttpContext.RequestAborted to become disposed leading to an ObjectDisposedException. This means that clearing out the CORS headers should be a non-issue since the client could have never received said headers over a closed connection in the first place.

If you are seeing a bunch of aborted requests and users need to keep refreshing the application to get a response, that points to a deeper issue. The ObjectDisposedException is merely a symptom of the already-aborted requests.

I would look at info-level and higher ASP.NET core logs to see if there's any indication why the connections associated with the failing requests are being closed.

[shirhatti]

@shirhatti I don't understand. For the first issue, when you say "This was fixed in CoreCLR.", what can I do to obtain that fix for my applications? When is that fix available for .NET Core 2.2 apps?

This was fixed in 2.2.1 which was released way back in June 2018

[jrmoreno1]

I have a website where I get this error (with 2.2.2) irregularly. Sometimes multiple times a day, sometimes several days without it happening. Running in process

Stacktrace:
at System.Threading.CancellationTokenSource.ThrowObjectDisposedException() at System.Threading.CancellationTokenSource.Cancel() at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.<>c__DisplayClass314_0.b__0(Object t)

[jkotalik]

@jrmoreno1 can you try running 3.0? This issue should be resolved there.

[gaborbuzasi]

@jrmoreno1 I can confirm that I had the same issue and this is resolved in 3.0.

I would still like to point out that there's so many breaking changes in 3.0 that it's not really an easy option for everyone to migrate just to resolve a pressing production issue.

[jrmoreno1]

@jkotalik : one of the frustrating things about this is it only happens in production, not on the available test servers. So, no, I haven’t tried an upgrade to 3.0. We are planning on skipping 3.0 and going to 3.1 as it is supposed to come out in November.

As an aside, you wouldn’t know if that was late or early November would you?

[sepehr1014]

@jkotalik : one of the frustrating things about this is it only happens in production, not on the available test servers. So, no, I haven’t tried an upgrade to 3.0. We are planning on skipping 3.0 and going to 3.1 as it is supposed to come out in November.

As an aside, you wouldn’t know if that was late or early November would you?

Same here. But the expected final ship date is December 2019 as Richard has mentioned in this post:
https://devblogs.microsoft.com/dotnet/announcing-net-core-3-1-preview-1/

[sepehr1014]

Isn't this fix going to be shipped in 2.2.8? At this stage it's quite hard to easily migrate to 3.0. (EF Core 3.0 disabling client validation being one of reasons)

[Tratcher]

There are two issues tracked in this thread.

1. CancellationTokenRegistration.Token throws an ODE if the associated CancellationTokenSource has been disposed. This was fixed in CoreCLR. This should be sufficient to prevent any process crashes.
   > This was fixed in 2.2.1 which was released way back in June 2018
2. There remains an issue where this a race between where the requestAborted token may be valid when you register it, but may have been disposed prior to using (due to a client disconnect). This results in an ODE which we catch and log as an application error, but it is benign.

We've re-worked the code to remove the chance of throwing an ODE in 3.0, however this doesn't meet the bar for a 2.2 backport since it's benign. We'd be happy to look into this again for 2.2 if anyone is actually experiencing a process crash due to this.

[jwfx]

Is there some workaround or way to prevent this from spilling into the application log?

We're using NLog together with ASP.NET Core 2.2.6 and currently this issue is spamming our logs and will potentially create customer support cases.

17:59:39.168 | ERROR | Connection id "0HLRD27EEMU5V", Request id "0HLRD27EEMU5V:00000002": An unhandled exception was thrown by the application. System.ObjectDisposedException: The CancellationTokenSource has been disposed. [...]