Aspnetcore: NullReferenceException in IISHttpContext

Created on 21 Dec 2018  ·  38Comments  ·  Source: dotnet/aspnetcore

Describe the bug

Since updating a website running in Azure App Service to use ASP.NET Core 2.2.0 with in-process hosting, I've observed a small number of NullReferenceExceptions being thrown via Application Insights.

System.NullReferenceException:
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeContext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer+<HandleRequest>d__27.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

To Reproduce

Application experiencing the issue was built from this commit: martincostello/website@ee2ca4ba02823b97062924b7dc8f313e6278a3f8

Expected behavior

HTTP request is processed successfully.

Screenshots

image

Additional context

Add any other context about the problem here.
Include the output of dotnet --info

area-servers bug servers-iis

Most helpful comment

If I understand this thread correctly, we are supposed to wait for another release and in the meantime this is not causing any harm. But it is really distracting to see these false positives where we expect to see only "real" exceptions. Is there some middleware that you can provide to catch this condition? I do not want to filter this on my side, for fear of eating up an authentic failure. TIA

All 38 comments

Do you know which request URL causes the issue?

I’ll dig further through AI later today for you to find out.

AI doesn't appear to have captured any other telemetry events for the specific exceptions (I guess it's happening too early in the request flow for managed code for any route matching to have run), but from looking at request logs, the best I can tell is that it was a request to the root of the website.

Same issue over here. Any workaround available ?
Are we sure this is related to the In/OutOfProcess option ?

For me this only started once I switched to in-process hosting after the roll out to Azure App Service of the hosting bundle was completed.

Do you have timestamps of these exceptions? Feels a bit like AppServices warmup request.

Four timestamps for these exceptions are:

  • 20/12/2018, 18:24:13
  • 21/12/2018, 00:37:07
  • 21/12/2018, 13:45:57
  • 25/12/2018, 15:38:01

@martincostello do you have Always On enabled?

In case you meant me, yes I do.

The frequency of requests there looks like it is coming from the app warmup request after IIS restarts from Always On. To confirm, can you turn off app warmup for a period and see if exceptions are still being thrown?

If that’s the case, the null ref is coming from the warmup request. This behavior is “fine” as the request was fake and it is supposed to startup Program.Main. However, we should more gracefully handle this type of request and not throw.

Get Outlook for iOShttps://aka.ms/o0ukef


From: Martin Costello notifications@github.com
Sent: Thursday, December 27, 2018 12:27 PM
To: aspnet/AspNetCore
Cc: Subscribed
Subject: Re: [aspnet/AspNetCore] NullReferenceException in IISHttpContext (#6075)

In case you meant me, yes I do.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faspnet%2FAspNetCore%2Fissues%2F6075%23issuecomment-450225392&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7Ce47a17cf79c14ed8965508d66c39c6a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636815392732641734&sdata=rXZLEMMAusc4ZBH47OXhlM1sQaNAzPMZuUSHa4a7H2w%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAH6uFVu8-uWuEVZkvZG2WobO1z3nwajcks5u9S1HgaJpZM4ZdnOe&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7Ce47a17cf79c14ed8965508d66c39c6a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636815392732651747&sdata=c%2BBvm7xm2865OIYE3l20Cp3c14zwpX9OMzwm%2BENkVWU%3D&reserved=0.

I've disabled _Always On_ and will leave it for a few days to see if it happens again, but it only happened a handful of times in the last 7 days as it is.

@jkotalik or we need to find the module that makes this request and make sure it sets pRawUrl.

@martincostello thank you for your help in figuring this out.

I have also the similar problem, with get_RemoteIpAddress

An unhandled exception has occurred while executing the request. System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeRemoteEndpoint()
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Microsoft.AspNetCore.Http.Features.IHttpConnectionFeature.get_RemoteIpAddress()
   at Microsoft.AspNetCore.Http.Internal.DefaultConnectionInfo.get_RemoteIpAddress()
   at Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware.ApplyForwarders(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)

Besides, my app was deployed at Azure app service with autoscale mode.

@qqbuby do you only see a single request failing per application startup?

@jkotalik I guess the error is coming from the app warmup request cause it doesn't happen frequently and I only see a few of error messages per day.

I talked to AppServices folks and this is not how "Always On" works. It's implemented using load balancer sending requests to the app.

I am seeing this error also localhost. Is it possibly related to the IIS Application Pool setting AlwaysRunning?

@bgiftge possibly, which variation of the error do you see? InitializeRemoteEndpoint or GetOriginalPath ?

hmm.. maybe I spoke too early. I was convinced that I saw the GetOriginalPath error a couple of times, but I am not finding it in my logs.

in case this helps, here some of our server logs.
EDIT: first request appears to _not_ overlap with the error.

25 Jan 2019 02:34:04.542 MyProject.Web Request GET / finished with status 404 in 52.0136ms 25 Jan 2019 02:32:11.237 MyProject.Web Request POST /api/xyz finished with status 200 in 15098.0613ms 25 Jan 2019 02:31:27.479 MyProject.Web Unexpected exception in IISHttpServer.HandleRequest. 25 Jan 2019 02:31:25.811 MyProject.Web MyProject.Web web host has started. 25 Jan 2019 02:31:13.429 MyProject.Web Starting MyProject.Web web host... 25 Jan 2019 02:31:13.411 MyProject.Web MyProject.Web web host successfully built. 25 Jan 2019 02:30:56.058 MyProject.Web Building MyProject.Web web host... 25 Jan 2019 02:30:56.005 MyProject.Web Configuring MyProject.Web web host build... 25 Jan 2019 02:30:39.169 MyProject.Web Injecting MyProject.Web secrets... 25 Jan 2019 02:30:38.061 MyProject.Web Initializing MyProject.Web...

This is caused by IIS ApplicationInitialization module that doesn't set pRawUrl on Win2012

App Service runs on 2016 these days though, right?

Sorry, I was wrong, this bug was fixed in RS2 update of win 10, AppService is on RS1

This is still happening in latest 2.2.2 runtime.
Using InProcess hosting with Application Pool Start Mode = AlwaysRunning

This wasn't fixed in 2.2.2. However, this was fixed in the latest 3.0-preview build of ANCM. If possible, please give that a try.

I'm having the same problem. As soon as I changed the hosting model to InProcess I started to get this error.
Object reference not set to an instance of an object. System.NullReferenceException at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60 {"method":"Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath","level":0,"assembly":"Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","line":0} {"method":"Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeContext","level":1,"assembly":"Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","line":0}

I didn't understand from the message exchange what's the solution for this Azure issue.

I'm glad this will be addressed in 3.0 but will this be addressed in 2.2.x or 2.x?

I'm glad this will be addressed in 3.0 but will this be addressed in 2.2.x or 2.x?

Were you able to fix this ? We have same issue but no work around in 2.2.2

Update: For us we had preload enabled flag turned on at the Website level. Disabling this fixed the error

@Rugved I wouldn't say that disabling IIS website preload fixes this issue, but its a valid workaround if you do not need/use preload.

image

What implications does this have on the running application? I am running 2.2 and need PreloadEnabled set to true. My app seems to be running without an issue even though I get this error when the application starts.

It shouldn't have any implication on the running application. It just gives a false positive for an exception.

If I understand this thread correctly, we are supposed to wait for another release and in the meantime this is not causing any harm. But it is really distracting to see these false positives where we expect to see only "real" exceptions. Is there some middleware that you can provide to catch this condition? I do not want to filter this on my side, for fear of eating up an authentic failure. TIA

+1

Same problem, Azure App Service, asp.net core 2.2
From what I understand - backporting is not happening because "2.2 out of support soon" aka we don't care enough ?
Given that .NET Core 3.0 is not LTS - and LTS release is only scheduled for November 2019 - some people prefer not to move to aspnetcore 3.0 yet. Would you estimate the effort for back porting the fix into 2.2 ?

I am on aspnetcore 3.0 and I still receive this. I am also utilizing InProcess hosting on an App Service.

@vsadams could you file another issue saying you are hitting this in 3.0?

Was this page helpful?
0 / 5 - 0 ratings