Aspnetcore: Regression in CPU utilisation with ASP.NET Core 2.2.0 web application

[martincostello]

Describe the bug

We have an ASP.NET Core web application that uses MVC to render Razor views that was recently updated from 2.1.6 to 2.2.0 which is showing semi-regular spikes in CPU utilisation where usage can increase by up to 4x for a short period before returning to baseline.

These spikes appear to occur semi-regularly, but inconsistently in time and across instances in a load-balanced fleet.

As series of screenshots that illustrate these CPU behaviours are included below.

The diff for the code and infrastructure changes between the two releases was to update all relevant NuGet package versions to 2.2.0 and to use CompatibilityVersion.Version_2_2, as well as installing the 2.2.0 runtime and Windows hosting pack.

The only other code change made was to use IHttpMessageHandlerFactory in a code path that is not in use for the environment configuration where we are observing this issue.

The application uses IIS out-of-process hosting (rather than the new default in-process mode) out of an abundance of caution pending a released fix for #4398 as this issue caused compatibility issues between the API this web application consumes and a separate application running ASP.NET 4.6.1 (see #4437).

In tandem with the above changes, the API that the web application consumes was also updated to use ASP.NET Core 2.2.0 in the same manner, and this application is not observing the same CPU utilisation changes despite its load being mirrored by the fact the web application depends on it.

This _naively_ leads me to the conclusion that there is a regression in ASP.NET Core somewhere in code paths related to Razor views (compared to say, APIs, controllers, model-binding, routing, runtime etc.), hence posting this issue here rather than in coreclr or corefx.

To Reproduce

Render Razor views with ASP.NET Core 2.2.0 on Windows using IIS out-of-process hosting for an extended period of time (more than ~1 hour to get a good chance of observing the CPU spike).

Expected behavior

Steady-state CPU utilisation of the application using 2.2.x should be comparable with 2.1.x.

Screenshots

Below are various Grafana chart screenshots that illustrate the issue in our production environment for the web application with commentary.

CPU spiking for a single EC2 instance

This graph shows the CPU usage of a specific EC2 instance of the web application during a peak period of traffic.

Zoom on a specific spike of the single EC2 instance

This graph zooms in on a specific spike from the chart above to show at a higher fidelity.

CPU spiking across EC2 fleet during the same time window

This graph shows the CPU usage of all the EC2 instances of the web application during the same peak period of traffic as the chart above.

Average CPU across the EC2 fleet during the same window (green line) with a -7 day comparison (yellow line)

This graph shows the average CPU usage of all the EC2 instances of the web application during the same peak period of traffic as the charts above.

Average CPU across the fleet for the last 7 days

This graph shows the average CPU usage of all the EC2 instances of the web application during the last seven days, which is the green line. The yellow line shows the same metric for 7 days previously.

The vertical red lines indicate code deployments. The second red line from the line indicates where the 2.2.0 version of the application was deployed. Subsequent lines are business-as-usual code deployments subsequent to the upgrade.

Note: The largest spikes come from CPU utilisation when new EC2 instances come into service from CPU-based auto-scaling and the application is installed onto the fresh instances. These spikes are expected and are separate from the issue being described here.

CPU utilisation per EC2 instance for the last 7 days

This graph is the same as the above, except for all EC2 instances, rather than the overall average of the fleet.

Average CPU across the fleet for the last 7 days for the underlying API

This graph shows the average CPU usage of all the EC2 instances of the API dependency of the web application during the last seven days, which is the green line. The yellow line shows the same metric for 7 days previously.

The vertical red lines indicate code deployments. The second red line from the line indicates where the 2.2.0 version of the application was deployed. Subsequent lines are business-as-usual code deployments subsequent to the upgrade.

This graph serves as a comparison baseline for a different application that is just a HTTP API that is also running 2.2.0, but is unaffected with the same CPU spiking.

Average CPU across the fleet for the last 7 days for the underlying API

This graph is the same as the above, except for all EC2 instances of the API, rather than the overall average of the fleet.

Additional context

Application runs on AWS EC2 c5.large instances using Windows Server 2012 R2.

More overall context for the application can be found in this blog post from back when we updated it from 2.0.x to 2.1.0.

As this is an internal line-of-business application that only appears to reproduce the issue under load, I cannot provide a repro for the issue. However, we're happy to provider any required telemetry/trace/dumps etc. that may help resolve the root cause to you privately.

[martincostello]

We're still investigating, but a preliminary look into things suggests that dotnet.exe is terminating/crashing and being recycled by IIS, which is the source of the CPU spikes and temporary slowdown in requests.

We're seeing warnings in the Windows Event Log that appear to correlate with the spikes:

A process serving application pool '{Application Name}' suffered a fatal communication error with the Windows Process Activation Service. The process id was '{Process ID}'. The data field contains the error number.

[martincostello]

Further investigation into our logs suggest a 1:1 correlation between CPU spikes and the above message being logged to the Windows Event Log.

It appears that dotnet.exe is terminating when used inside IIS with OutOfProcess hosting at seemingly random intervals. When it is terminated, IIS re-launches dotnet.exe, which causes the CPU spike due to all the process startup code running.

The pause for the process to start and initialize looks to take ~5 seconds, where requests queue and IIS connections spike up while waiting for the new process to be ready to service requests.

[martincostello]

We've having a different issue with in-process hosting with the same application (that I'm still investigating which is an UriFormatException from somewhere being logged by the IISHttpServer logger), which might mean that we can't upgrade this application to ASP.NET 2.2.0 at all as there's issues for us with both hosting models.

[martincostello]

The above comment was actually a latent bug in some existing code that wasn't appearing in the logs because of #4206 when in-process hosting was enabled.

[jkotalik]

@martincostello few questions:

• In the site's web.config, are you using AspNetCoreModule or AspNetCoreModuleV2?
• Are you using AlwaysRunning/preloadEnabled?

If you could get a dump of the w3wp process when it crashes and send it to my personal email, we can take a look into it on our side.

[martincostello]

I'll look at reverting back to out-of-process next week so I can generate a crash dump as we've switched over to in-process for the application this week.

Relevant parts of web.config:

<handlers>
  <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
</handlers>
<aspNetCore requestTimeout="00:01:00" disableStartUpErrorPage="true" processPath="dotnet" arguments=".\MyApp.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="OutOfProcess" />

Relevant parts of applicationHost.config:

<applicationPools>
  <add name="myapp" autoStart="true" managedRuntimeVersion="" startMode="AlwaysRunning">
</applicationPools>

<sites>
  <site name="myapp" id="1">
    <application path="/" applicationPool="myapp">
      <virtualDirectory path="/" physicalPath="c:\myapp" />
    </application>
  </site>
</sites>

[martincostello]

Again a process dump will be some time next week, but I've just found this in our event log that may help. It appears to happen at the same time as the event log writes an error about the process crash and restart. In this case, it's from in-process hosting.

Caption: w3wp.exe - System Error
Channel: System
EventID: 26
EventType: INFO
Message: Application popup: w3wp.exe - System Error : A new guard page for the stack cannot be created. 
SourceName: Application Popup

[martincostello]

I'm starting to wonder if maybe this crash coupled with #6332 are hinting at a stack overflow somewhere that's just manifesting in different ways?

[davidfowl]

That sounds likely. The stack size is a lot smaller under IIS (in proc). It’s soethifn we’re trying to address.

[jkotalik]

Try to increase the stack size of w3wp by doing: EDITBIN /STACK:SIZE_IN_BYTES w3wp.exe.

[martincostello]

@jkotalik To try this out with our production boxes (which is the only place we can seem to reliably reproduce this) we'd need to run EDITBIN as part of the deployment steps onto our EC2 instances.

I see EDITBIN is something that comes with Visual Studio though, is there an alternate way to apply this to w3wp.exe at deloyment time?

[jkotalik]

I didn't see any other way through google searching. @pakrym IIRC there may be a reg key to edit the stack size.

My gut we could be hitting the max stack size due to calling stackalloc here: https://github.com/aspnet/AspNetCore/blob/master/src/Servers/IIS/IIS/src/Core/IO/AsyncWriteOperationBase.cs#L36.

This could lead to around 1KB on the stack per request. The stack size by default is 256/512KB (depending on bitness) which seems like a bottleneck.

[pakrym]

Unfortunately EDITBIN is the only way I know to change w3wp stack size.

My gut we could be hitting the max stack size due to calling stackalloc here: https://github.com/aspnet/AspNetCore/blob/master/src/Servers/IIS/IIS/src/Core/IO/AsyncWriteOperationBase.cs#L36.

This could lead to around 1KB on the stack per request. The stack size by default is 256/512KB (depending on bitness) which seems like a bottleneck.

I doubt thats it, it's a 1KB per 256KB stack.

[martincostello]

Unfortunately these process crashes mean that both the in and out-of-process hosting for 2.2.0 are too unstable for production usage for one of our web applications, so we'll have to revert to 2.1.x and await appropriate fix(es).

Once we've rolled-back, I'll look at seeing if the V1 hosting module for out-of-process shows the same problems for us or not. That should at least narrow things down between V2 hosting and something in 2.2.0 generally.

[martincostello]

FYI, we're re-trying 2.2.0 out again with the following csproj settings now to see if we can narrow down whether it's (ASP).NET Core 2.2.0 we're having trouble with, or the new V2 IIS hosting.

<AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel>
<AspNetCoreModuleName>AspNetCoreModule</AspNetCoreModuleName>

[jkotalik]

Sounds good. Thanks again for the through investigation.

[martincostello]

We've now been re-running this application with 2.2.0 using the V1 ANCM module for over 24 hours in production, and we've not yet seen any recurrence of process crashes/restarts or elevated CPU usage .

We'll be monitoring this further through until the start of next week, but that would seem to suggest the cause of our issues is something in ANCM V2.

[muratg]

@martincostello Are you able to capture a crash dump w3wp crashing when ANCM V2 + .NET Core 2.2 + Out of proc combination is used?

[martincostello]

I’ll try to capture one again next week. We did try this week but they don’t seem to be being generated on our EC2 instances: https://stackoverflow.com/q/54146650/1064169

[shirhatti]

@martincostello Have you had any success capturing a process dump?

[martincostello]

Unfortunately, not yet. Something on our EC2 instances is causing the crashes just to cause a System Error pop-up rather than cause WER to be invoked, meaning that the crashes aren’t creating dumps.

I’m still trying to isolate what’s causing this so I can disable it and then get hold of a dump file.

[martincostello]

We're in the process of doing a Server 2012 R2 to Server 2016 migration, and the scripts I have work there. As a fallback I'll wait for that migration to complete for the application in question, then I should be able to get a dump with the crashing configuration.

[martincostello]

An update on this, we're planning to roll-out the upgrade to Windows Server 2016 next week, after which I should be able to temporarily revert the downgrade to ANCM v1 in order to capture a crash dump for analysis.

[martincostello]

Further update: we've completed our migration to Windows Server 2016. We're in a change freeze until Monday 18th now, so we're aiming to run the "buggy" version for a short period next week so we can get a crash dump to send over for analysis.

[martincostello]

We've now temporarily deployed a build with ANCM v2 enabled for out-of-process. Just waiting for the issue to repro and generate a crash dump.

[martincostello]

@jkotalik I've captured 2 crash dumps from w3wp.exe now using ANCM v2 with out-of-process hosting. Can you confirm where I should send them (they're ~75MB each).

[jkotalik]

Send them to my email listed on my github profile. Thanks!

[jkotalik]

The dumps also may be too large for email upload. Just email me and we start the process.

[martincostello]

Email sent.

[martincostello]

Dump files have been shared with @jkotalik and @pakrym.

[shapeh]

Just chipping in here. Getting the same the error here:

A process serving application pool 'www.example.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4952'. The data field contains the error number.

Using 2.2.2, Win2016 and in-process, and IIS restarts the app-pool irregular. I am running DebugDiag with a breakpoint on Ntdll!ZwTerminateProcess but it is not capturing any dumps.

@martincostello - can you let me know how you capture the dump on your side. I would be happy share my log as well.

[aspnetcorev2.dll] Current process bitness type detected as isX64=1
[aspnetcorev2.dll] Processing entry 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2.dll] Binary type 6
[aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Filesdotnet\host\fxr\2.2.2\hostfxr.dll'
[aspnetcorev2.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com.\example.dll'
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Filesdotnetdotnet.exe' hostfxr path: 'C:\Program Filesdotnet\host\fxr\2.2.2\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2.dll] Argument[1] = 'C:\inetpub\sites\www.example.com.\example.dll'
[aspnetcorev2.dll] Argument[2] = 'exec'
[aspnetcorev2.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll'
[aspnetcorev2.dll] c:\b\w\da744fbcc13abce\src\iisintegration\src\aspnetcoremodulev2\commonlib\fileoutputmanager.cpp:142 Operation failed with LastError: 32 HR: 0x80070020
[aspnetcorev2.dll] Loading request handler: 'C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App\2.2.2\aspnetcorev2_inprocess.dll'. Process Id: 5588.. File Version: 12.2.19024.2. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 522705f9a27b99ca4ad261f2e89fe51a77b2338e.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\example.dll exec "C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll"' path: 'C:\inetpub\sites\www.example.com\'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Filesdotnet\host\fxr\2.2.2\hostfxr.dll'
[aspnetcorev2_inprocess.dll] Converted argument '.\example.dll' to 'C:\inetpub\sites\www.example.com.\example.dll'
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Filesdotnetdotnet.exe' hostfxr path: 'C:\Program Filesdotnet\host\fxr\2.2.2\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Filesdotnetdotnet.exe'
[aspnetcorev2_inprocess.dll] Argument[1] = 'C:\inetpub\sites\www.example.com.\example.dll'
[aspnetcorev2_inprocess.dll] Argument[2] = 'exec'
[aspnetcorev2_inprocess.dll] Argument[3] = 'C:\Web\example\example\bin\Release\netcoreapp2.2\example.dll'
[aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'c:\windows\system32\inetsrv'
[aspnetcorev2_inprocess.dll] Setting dll directory to c:\windows\system32\inetsrv
[aspnetcorev2_inprocess.dll] Setting current directory to C:\inetpub\sites\www.example.com\
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'C:\inetpub\sites\www.example.com\' started the coreclr in-process successfully.'
End Event Log Message.
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\sites\www.example.com\'
[aspnetcorev2_inprocess.dll] Starting file watcher thread

.NET Core SDK (reflecting any global.json):
Version: 2.2.104
Commit: 73f036d4ac

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Filesdotnet\sdk\2.2.104\

Host (useful for support):
Version: 2.2.2
Commit: a4fd7b2c84

.NET Core SDKs installed:
2.1.300 [C:\Program Filesdotnet\sdk]
2.1.301 [C:\Program Filesdotnet\sdk]
2.2.100 [C:\Program Filesdotnet\sdk]
2.2.104 [C:\Program Filesdotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.1 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.2 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.3 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.4 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.1 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.2 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.1 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.3 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.4 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.1 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.2 [C:\Program Filesdotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.1 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.4 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.1 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.2 [C:\Program Filesdotnet\shared\Microsoft.NETCore.App]

[martincostello]

@shapeh We capture the dumps like this: https://stackoverflow.com/q/54146650/1064169 - I never go to the bottom of why it didn't work on our 2012 R2 EC2 instances, but it works fine on our 2016 ones.

[martincostello]

@pakrym @jkotalik If the fix for this underlying issue was indeed add in 3.0.0 preview-2, would it be a candidate for porting to the 2.2.x branch for inclusion with 2.2.3 (or 2.2.4?).

[pakrym]

@pakrym @jkotalik If the fix for this underlying issue was indeed add in 3.0.0 preview-2, would it be a candidate for porting to the 2.2.x branch for inclusion with 2.2.3 (or 2.2.4?).

Yes.

[shapeh]

Not sure if this is related to this bug but I have been running Process Monitor on w3wp.exe and are getting 1000's of PATH NOT FOUND when using ResponseCaching.

w3wp.exe is calling CreateFile several 1000 times for methods I have marked in my ASP.NET Core project with something like this:

[ResponseCache(Duration = 5, VaryByQueryKeys = new string[] { "country", "id" })] // ResponseCaching
[Route("~/{lang}/database/{country}/{letter}/{title}-{id}", Name = "database-country")]

If I am not mistaken, it seems like ResponseCaching is trying to create cache files in the the actual web folder root. In my case, root is c:\inetpub\wwwroot\www.example.com

This is what ProcMon returns:

"Time of Day","Process Name","PID","Operation","Path","Result","Detail"
"5:12:25.8208570 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo","NAME NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8210456 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8213898 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8215363 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8216481 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8217366 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8218331 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8219342 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8220240 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8221208 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8222883 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\my-title-1234","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"
"5:12:25.8224565 AM","w3wp.exe","8144","CreateFile","C:\inetpub\sites\www.example.com\fo\database\belgien\m\my-title-1234\web.config","PATH NOT FOUND","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a"

Since these paths do not exists (and should not), procmon reports 1000s of "PATH NOT FOUND" in very short time. Perhaps this is causing restarts?

Update:
It seems to happen with many other pages requested (MVC), e.g.
C:\inetpub\sites\www.example.com\Views\Shared\_TagsPartial.sv-SE.cshtml

• this file do not exist, and does not even follow naming scheme of the project.

Maybe this is related to https://github.com/aspnet/AspNetCore/issues/4206#issuecomment-445612167

My Program.cs

    public static class Program
    {
        public static void Main(string[] args)
        {
            CurrentDirectoryHelpers.SetCurrentDirectory(); // asp.net core 2.2 hack. In 3.0 this will be fixed: https://github.com/aspnet/AspNetCore/issues/4206#issuecomment-445612167
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
           WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>();
    }

[martincostello]

@shapeh Are you running 2.2.0 or 2.2.1?

In those versions the current directory of the app was changed to be that of IIS itself, rather than the location of the app content itself. The previous behaviour was restored in 2.2.2.

If you’re not running 2.2.2, try upgrading to the latest release and see if that fixes it.

[shapeh]

@martincostello I am running 2.2.2 - so if I reading you correct, I should try and remove the
CurrentDirectoryHelpers.SetCurrentDirectory(); and see if that solves the issue?

[martincostello]

@shapeh Yeah, you shouldn't need to do that anymore with 2.2.2. If that doesn't fix it then it's probably a different problem, and I'll defer to the ASP.NET team on how to resolve that.

[shapeh]

@martincostello - I will try that and see what happens. Out of curiosity, have you tried running 2.2.2 on win2016 or are you sticking to 2.2.0 win2016 for now?

[shapeh]

Just tried removing CurrentDirectoryHelpers.SetCurrentDirectory(); - no luck. Still get 1000's of PATH NOT FOUND in a few minutes. I have turned off any ResponseCaching too.

[pakrym]

@shapeh it seems like you are having a different issue. Can you please open a new bug so we don't sidetrack this one.

[shapeh]

@pakrym - okay, definitely will do that.

[martincostello]

@pakrym @jkotalik If the fix for this underlying issue was indeed add in 3.0.0 preview-2, would it be a candidate for porting to the 2.2.x branch for inclusion with 2.2.3 (or 2.2.4?).

Yes.

Has the underlying fix for this been ported to 2.2 for servicing yet?

[martincostello]

Was this related to https://github.com/aspnet/Announcements/issues/352?

[anurse]

Yep, this should be resolved now.