Since updating a website running in Azure App Service to use ASP.NET Core 2.2.0 with in-process hosting, I've observed a small number of NullReferenceException
s being thrown via Application Insights.
System.NullReferenceException:
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeContext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer+<HandleRequest>d__27.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Application experiencing the issue was built from this commit: martincostello/website@ee2ca4ba02823b97062924b7dc8f313e6278a3f8
HTTP request is processed successfully.
Add any other context about the problem here.
Include the output of dotnet --info
Looking at the code in master, it would seem that GetRawUrlInBytes()
saw NativeRequest->pRawUrl
to be null or empty, and returned null
, which then isn't handled by GetOriginalPath()
.
Do you know which request URL causes the issue?
I’ll dig further through AI later today for you to find out.
AI doesn't appear to have captured any other telemetry events for the specific exceptions (I guess it's happening too early in the request flow for managed code for any route matching to have run), but from looking at request logs, the best I can tell is that it was a request to the root of the website.
Same issue over here. Any workaround available ?
Are we sure this is related to the In/OutOfProcess option ?
For me this only started once I switched to in-process hosting after the roll out to Azure App Service of the hosting bundle was completed.
Do you have timestamps of these exceptions? Feels a bit like AppServices warmup request.
Four timestamps for these exceptions are:
20/12/2018, 18:24:13
21/12/2018, 00:37:07
21/12/2018, 13:45:57
25/12/2018, 15:38:01
@martincostello do you have Always On
enabled?
In case you meant me, yes I do.
The frequency of requests there looks like it is coming from the app warmup request after IIS restarts from Always On. To confirm, can you turn off app warmup for a period and see if exceptions are still being thrown?
If that’s the case, the null ref is coming from the warmup request. This behavior is “fine” as the request was fake and it is supposed to startup Program.Main. However, we should more gracefully handle this type of request and not throw.
Get Outlook for iOShttps://aka.ms/o0ukef
From: Martin Costello notifications@github.com
Sent: Thursday, December 27, 2018 12:27 PM
To: aspnet/AspNetCore
Cc: Subscribed
Subject: Re: [aspnet/AspNetCore] NullReferenceException in IISHttpContext (#6075)
In case you meant me, yes I do.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faspnet%2FAspNetCore%2Fissues%2F6075%23issuecomment-450225392&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7Ce47a17cf79c14ed8965508d66c39c6a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636815392732641734&sdata=rXZLEMMAusc4ZBH47OXhlM1sQaNAzPMZuUSHa4a7H2w%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAH6uFVu8-uWuEVZkvZG2WobO1z3nwajcks5u9S1HgaJpZM4ZdnOe&data=02%7C01%7CJustin.Kotalik%40microsoft.com%7Ce47a17cf79c14ed8965508d66c39c6a7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636815392732651747&sdata=c%2BBvm7xm2865OIYE3l20Cp3c14zwpX9OMzwm%2BENkVWU%3D&reserved=0.
I've disabled _Always On_ and will leave it for a few days to see if it happens again, but it only happened a handful of times in the last 7 days as it is.
@jkotalik or we need to find the module that makes this request and make sure it sets pRawUrl
.
@martincostello thank you for your help in figuring this out.
I have also the similar problem, with get_RemoteIpAddress
An unhandled exception has occurred while executing the request. System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeRemoteEndpoint()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Microsoft.AspNetCore.Http.Features.IHttpConnectionFeature.get_RemoteIpAddress()
at Microsoft.AspNetCore.Http.Internal.DefaultConnectionInfo.get_RemoteIpAddress()
at Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware.ApplyForwarders(HttpContext context)
at Microsoft.AspNetCore.HttpOverrides.ForwardedHeadersMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
Besides, my app was deployed at Azure app service with autoscale mode.
@qqbuby do you only see a single request failing per application startup?
@jkotalik I guess the error is coming from the app warmup request cause it doesn't happen frequently and I only see a few of error messages per day.
I talked to AppServices folks and this is not how "Always On" works. It's implemented using load balancer sending requests to the app.
I am seeing this error also localhost. Is it possibly related to the IIS Application Pool setting AlwaysRunning
?
@bgiftge possibly, which variation of the error do you see? InitializeRemoteEndpoint or GetOriginalPath ?
hmm.. maybe I spoke too early. I was convinced that I saw the GetOriginalPath error a couple of times, but I am not finding it in my logs.
in case this helps, here some of our server logs.
EDIT: first request appears to _not_ overlap with the error.
25 Jan 2019 02:34:04.542
MyProject.Web Request GET / finished with status 404 in 52.0136ms
25 Jan 2019 02:32:11.237
MyProject.Web Request POST /api/xyz finished with status 200 in 15098.0613ms
25 Jan 2019 02:31:27.479
MyProject.Web Unexpected exception in IISHttpServer.HandleRequest.
25 Jan 2019 02:31:25.811
MyProject.Web MyProject.Web web host has started.
25 Jan 2019 02:31:13.429
MyProject.Web Starting MyProject.Web web host...
25 Jan 2019 02:31:13.411
MyProject.Web MyProject.Web web host successfully built.
25 Jan 2019 02:30:56.058
MyProject.Web Building MyProject.Web web host...
25 Jan 2019 02:30:56.005
MyProject.Web Configuring MyProject.Web web host build...
25 Jan 2019 02:30:39.169
MyProject.Web Injecting MyProject.Web secrets...
25 Jan 2019 02:30:38.061
MyProject.Web Initializing MyProject.Web...
This is caused by IIS ApplicationInitialization module that doesn't set pRawUrl on Win2012
App Service runs on 2016 these days though, right?
Sorry, I was wrong, this bug was fixed in RS2 update of win 10, AppService is on RS1
This is still happening in latest 2.2.2 runtime.
Using InProcess hosting with Application Pool Start Mode = AlwaysRunning
This wasn't fixed in 2.2.2. However, this was fixed in the latest 3.0-preview build of ANCM. If possible, please give that a try.
I'm having the same problem. As soon as I changed the hosting model to InProcess I started to get this error.
Object reference not set to an instance of an object.
System.NullReferenceException at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath
Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
{"method":"Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.GetOriginalPath","level":0,"assembly":"Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","line":0}
{"method":"Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.InitializeContext","level":1,"assembly":"Microsoft.AspNetCore.Server.IIS, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60","line":0}
I didn't understand from the message exchange what's the solution for this Azure issue.
I'm glad this will be addressed in 3.0 but will this be addressed in 2.2.x or 2.x?
I'm glad this will be addressed in 3.0 but will this be addressed in 2.2.x or 2.x?
Were you able to fix this ? We have same issue but no work around in 2.2.2
Update: For us we had preload enabled flag turned on at the Website level. Disabling this fixed the error
@Rugved I wouldn't say that disabling IIS website preload fixes this issue, but its a valid workaround if you do not need/use preload.
What implications does this have on the running application? I am running 2.2 and need PreloadEnabled set to true. My app seems to be running without an issue even though I get this error when the application starts.
It shouldn't have any implication on the running application. It just gives a false positive for an exception.
If I understand this thread correctly, we are supposed to wait for another release and in the meantime this is not causing any harm. But it is really distracting to see these false positives where we expect to see only "real" exceptions. Is there some middleware that you can provide to catch this condition? I do not want to filter this on my side, for fear of eating up an authentic failure. TIA
+1
Same problem, Azure App Service, asp.net core 2.2
From what I understand - backporting is not happening because "2.2 out of support soon" aka we don't care enough ?
Given that .NET Core 3.0 is not LTS - and LTS release is only scheduled for November 2019 - some people prefer not to move to aspnetcore 3.0 yet. Would you estimate the effort for back porting the fix into 2.2 ?
I am on aspnetcore 3.0 and I still receive this. I am also utilizing InProcess hosting on an App Service.
@vsadams could you file another issue saying you are hitting this in 3.0?
Most helpful comment
If I understand this thread correctly, we are supposed to wait for another release and in the meantime this is not causing any harm. But it is really distracting to see these false positives where we expect to see only "real" exceptions. Is there some middleware that you can provide to catch this condition? I do not want to filter this on my side, for fear of eating up an authentic failure. TIA