Aspnetcore: NullReferenceException thrown by IISHttpContext.AbortIO in ASP.NET Core 3.0.1
Describe the bug
From a weekly review of our production error logs, we noticed a single isolated example of the following exception from an ASP.NET Core 3.0.1 app running on Windows Server 2016 with IIS in-process hosting:
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.AbortIO(Boolean clientDisconnect)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnDisconnect(IntPtr pvManagedHttpContext)
To Reproduce
Unknown - presumably some edge-case when the client disconnects.
From looking at the code, I'd guess that _somehow_ _bodyOutput was null.
martincostello
All 46 comments
@martincostello to be clear, are you seeing any user-visible problems from this or is it ust a spurious exception in the logs?
anurse
on 3 Dec 2019
I鈥檒l double-check tomorrow, but I think it鈥檚 just spurious.
martincostello
on 3 Dec 2019
a single isolated example
Given that it's one occurrence, it may be a lowish priority but we'll look and see what we can find. Disconnect races do happen.
anurse
on 3 Dec 2019
Yeah this is a curious one. _bodyOutput is initialized in the constructor for the IISHttpContext. I don't see what else could be throwing here for a null reference... Getting line numbers would be nice 馃槃
jkotalik
on 3 Dec 2019
@anurse I've correlated the error log with our IIS logs, and there appears to have been no user-facing impact (I can only see 200 and 302 HTTP statuses 1 minute either side of the exception being logged).
martincostello
on 3 Dec 2019
I can confirm this error. It occurs to us quite often when working from Visual Studio. The error is not user-facing, but quite annoying through. In the call stack there is just:
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnDisconnect(IntPtr pvManagedHttpContext)
Since this is from Visual Studio, it is running IISExpress. Building for netcoreapp2.2. In the log I see that .NET Core app version 2.2.7 is loaded.
PatrickHofman
on 3 Dec 2019
I had a quick peek around in the code, and it doesn't look like the field is set (at least directly) in the constructor, but here in InitializeContext():
Which gets called by ProcessRequestAsync() here:
Maybe somehow the disconnect happened before the processing started?
martincostello
on 6 Dec 2019
I am also facing this error quote often! My project is in .net core 3.1
bhrugen
on 19 Mar 2020
@bhrugen Do you have a repro handy?
shirhatti
on 20 Mar 2020
I am also getting this error on core 3.1 - I have my code on a repo if needed to be seen. Happens once and then if I continue my program it does not happen again.
DanielRoberts21
on 29 Mar 2020
Hi Team. Any update on this issue. After updating my visual studio professional 2019 version to 16.5.2, i am facing this issue quite often.
harikrishnakada
on 3 Apr 2020
Let's bump the priority of this issue; more people are hitting it than we originally believed. Depending on severity as well, we may backport.
jkotalik
on 4 Apr 2020
I can just confirm it.
After upgrade to VS to 16.5.2, during debug using IIS Express, Using .net core 3.1.3. and Blazor.
I see this error a lot.
Sometimes debug starts, but sometimes not, let's say it on 5 starts there is 1 exception.
It's quite anoyoing.
TecumStricto
on 6 Apr 2020
This happens to me as well. context becomes null for some reason.
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnDisconnect
klanglie
on 6 Apr 2020
This happens to me as well. context becomes null for some reason.
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnDisconnect
I have been having this issue all day, driving me nuts.
This started to happen after I updated Visual Studio to version 16.5.2.
Project: Angular 9 SPA project with .net core 3.1
donet --version outputs 3.1.201
What eventually fixed it for me was installing .net core 3.1.3 runtime on my machine. Now I can't reproduce the error (was quite constant before installing it)
GisliSig
on 6 Apr 2020
This started to happen after I updated Visual Studio to version 16.5.2.
Project: Angular 9 SPA project with .net core 3.1
donet --version outputs 3.1.201
This seems to be getting a huge uptick in hits over the past few days, specifically around 16.5.2.
What eventually fixed it for me was installing .net core 3.1.3 runtime on my machine. Now I can't reproduce the error (was quite constant before installing it)
I don't know necessarily know how the 3.1.3 runtime had a fix for this issue. There weren't any issues fixed in ANCM/IIS in 3.1.3. Can you continue to monitor to see if you rehit this issue?
jkotalik
on 6 Apr 2020
Hey, can any folks on this thread check the file version of iisexpress.exe your app is using? It should be possible to find the path via Task Manager, when looking at the process that is running your app, then you can view the Properties for the exe itself in File Explorer to get the File Version Info. We have a suspicion that this may have actually been a change to IIS Express, rather than ASP.NET Core, so getting accurate version info would help.
anurse
on 6 Apr 2020
@shirhatti did IIS Express update in VS 16.5?
anurse
on 6 Apr 2020
I don't know necessarily know how the 3.1.3 runtime had a fix for this issue. There weren't any issues fixed in ANCM/IIS in 3.1.3. Can you continue to monitor to see if you rehit this issue?
I tried to reproduce this and sure enough, it came up again. I uninstalled "Microsoft ASP.NET Core 3.1.3 - Shared Framework" that I installed today and when it was way easier to reproduce almost constantly.
GisliSig
on 6 Apr 2020
It looks like IISExpress updated in VS 16.5.
To confirm, if you are running VS 16.4, this doesn't repro?
jkotalik
on 6 Apr 2020
@anurse IIS version I am running is 10.0.18362.1
DanielRoberts21
on 7 Apr 2020
@jkotalik I tried on my home station to reproduce this bug, Had installed Visual Studio version 16.4.4. Did not manage to reproduce this bug with the same project as on my work station.
Upgraded to 16.5.2 and managed to reproduce it once.
Went back on my work station, I uninstalled and did a fresh install.
I did notice in the debug window that there were a lot of TaskCanceledException appearing. Noticed that I had "Enable Just My Code" disabled.
I enabled it and I haven't been able to reproduce after that change.
I have had "Enable Just My Code" disabled for a long time now, but I have only seen this error after I updated to 16.5.2. Before that time I had 16.4 and I never got this error.
GisliSig
on 7 Apr 2020
VS Enterprise 2019 16.5.1
IIS Express 10.0.18362.1
Dotnet core 3.1.2
EDIT: Having same issues since vs update to 16.5.1
kennyfowler
on 7 Apr 2020
Hey, can any folks on this thread check the file version of
iisexpress.exeyour app is using? It should be possible to find the path via Task Manager, when looking at the process that is running your app, then you can view the Properties for the exe itself in File Explorer to get the File Version Info. We have a suspicion that this may have actually been a change to IIS Express, rather than ASP.NET Core, so getting accurate version info would help.
No idea how to find this in Task Manager, sorry. Can you provide more detail on how to find the process in question?
capesean
on 8 Apr 2020
If your site is running in IIS Express, there should be an item in the Details tab of Task Manager for iisexpress.exe. Right click it and choose "Properties". Then share the "File version" entry from the "Details" tab (I was wrong about File Explorer, there's no need to look there; the Task Manager properties page has all the info):
If you can't find iisexpress.exe there, make sure your app is running and you are running in IIS Express.
If you're running VS 16.5.1, I'd expect this to be 10.0.18362.1 (as others in this thread indicate). I think we've identified that this was a change in IIS Express though, we just need to track down what changed and how it broke us :).
anurse
on 8 Apr 2020
Sorry, didn't know it was on the Details tab.
10.0.18362.1
capesean
on 8 Apr 2020
No problem! I wasn't clear before :).
anurse
on 8 Apr 2020
Some updates here, after triaging.
We believe we have some thoughts on how this can occur. In general, the _bodyOutput value shouldn't be null when the disconnect handler runs, but there may be a race caused by a client quickly disconnecting which means the disconnect handler runs before the connect handler finishes all it's work. We're going to look in to making that logic more defensive to avoid these errors.
The original report was in an older version of IIS, so we know that the race existed before, but it didn't seem to be very common. There doesn't appear to be any change in the IIS Express version in 16.5 that would cause this problem, but it could be that other changes just adjusted the timings enough to make this race more likely to occur.
Is anyone seeing a functional issue (i.e. one that would impact your users) because of this? Our current understanding is that it's happening after the client disconnects, so there shouldn't be a functional problem. Even if this isn't a functional issue, we still plan to fix it, but understanding if this is affecting your users or just showing up in production error logs or at dev time helps us prioritize appropriately.
anurse
on 8 Apr 2020
As far as I've seen the issue, it's just an issue when debugging as it pauses the execution of the application, from a user perspective, nothing goes wrong.
DanielRoberts21
on 8 Apr 2020
it's just an issue when debugging as it pauses the execution of the application, from a user perspective
I'm curious, do you have the debugger configured to break on first-chance exceptions? This exception should be being caught further up the call stack, unless you have configured the debugger to break on first-chance exceptions I'd expect the debugger not to break.
anurse
on 8 Apr 2020
I haven't changed the debuggers configuration at all so it will be whatever the default settings for it are. I'm not really sure where the configuration for it is
DanielRoberts21
on 8 Apr 2020
Maybe that's what happened in VS16.5.2, the debugger settings changed?
Because I've also noticed now when I hit an error in debug, then if I press F5, it stops a number of times thereafter, I have to F5 about 5-10 times before I get to the point where it would return a 500 error (or whatever).
I haven't looked into what changed or how to revert a setting(?) to go back to my previous debugging experience, but it could be related?
capesean
on 8 Apr 2020
Maybe. What do you all have showing in the "Exception Settings" tool window? Any exception checked in that list will cause the debugger to break as soon as it's thrown. If the exception isn't checked, it shouldn't cause the debugger to break unless it goes uncaught. You can filter this list using a search box in the top-right of the tool window. For example, I see the following for exceptions with "null" in the name:
None of the .NET-related exceptions are checked.
anurse
on 8 Apr 2020
I guess I should take off the NullReferenceException ?
capesean
on 8 Apr 2020
Just hit it again
capesean
on 8 Apr 2020
I guess I should take off the NullReferenceException ?
Well, that is what's causing you to break into the debugger on this exception. You could also check that box in your second screenshot that says "Except when thrown from Microsoft.AspNetCore.Server.IIS" if you still want the debugger to break when a NullRef is thrown (as opposed to being uncaught) but not in this specific case.
anurse
on 9 Apr 2020
Thanks! I really need to brush up on my debugger understanding....
capesean
on 9 Apr 2020
Even after VS 2019 update to latest one16.5.3, error is still there.
IIS Express is version, 10.0.18362.1, 17.01.2020 16:02
TecumStricto
on 9 Apr 2020
Even after VS 2019 update to latest one16.5.3, error is still there.
Yep, we haven't made a fix yet, so that would make sense :). Your exception settings also seem to be configured to break when this exception is thrown. I have the same suggestion for you as I described above. You should check the box in the Exception Thrown popup that says "Except when thrown from Microsoft.AspNetCore.Server.IIS". When we make a fix, you won't need that, but it's relatively safe to do so now if you aren't seeing a functional issue because of this.
anurse
on 9 Apr 2020
I am also hitting this issue with ASP.NET Core 3.1. I have been able to reproduce this exact issue on a fairly consistent basis. I wrote about it on StackOverflow. Basicallly, executing a RequestDelegate from an Endpoint which calls Response.Redirect seems to hit this on a somewhat random frequency which makes it appear to be some sort of race condition.
It seems to be the exact same issue that used to exist under ASP.NET Framework that over time most developers became aware of, also caused by Response.Redirect, the famed Thread Abort Exception.
jscarle
on 30 Apr 2020
Here's a stack trace that could help:
[ConnectionResetException: The client has disconnected]
at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult(Int16 token)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.ReadForm()
at BasePage.get_Form()
at PostBackPage.Initialize()
at BasePage.ProcessRequest(HttpContext context)
at EndpointInformation`1.Page[T](HttpContext context)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
I also have this issue: System.NullReferenceException | HResult=0x80004003 | Message=Object reference not set to an instance of an object.
This error occurred _only if I comment_ the following line:
app.UseSpaStaticFiles(StaticFileOptions(env)); (ignore the internal StaticFileOptions(env) method.
Well, the check for null is a nice addition to the code. So, I submitted a pull request above (#21858).
aaltotsky
on 15 May 2020
Same behavior here, IISHttpContext becomes null for some reason
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer.OnDisconnect
Visual Studio 16.5.5 and Asp Net Core 3.1 application with Razor Page.
Problem occurs very often in debugging the application on VS.
engineering87
on 19 May 2020
VS 16.6.0 Preview 6.0
IIS 10.0.18362.1
Net Core 3.1.3
For me it just started happening. I'm not sure if it was I added Areas to my project, maybe external authentication. or upgraded to the latest Preview 6.0. But you already have a good list if cases here, seems you are hot on the trail.
joebiker
on 29 May 2020
This may sound silly, something that worked nicely for me was using dotnet run instead of using the IIS Project from the debugger
fredyfx
on 31 May 2020
BrennanConroy
on 11 Jun 2020
Related issues
Kevenvz
路
3Comments
aurokk
路
3Comments
groogiam
路
3Comments
guardrex
路
3Comments
TanvirArjel
路
3Comments
Most helpful comment
I am also facing this error quote often! My project is in .net core 3.1