Runtime: EF 6.1.3 Crashing IIS Express with FatalExecutionEngineError in ASP.Net Core MVC

[adam8797]

I (and a few other users) have been experiencing a problem when using EF 6.1.3 and IIS Express. For some reason, sites will crash very inconsistently. Some times, it works, other times it does not.

For the sake of not being redundant, I'll link to a StackOverflow question I put up on the subject, where I go into more detail. Other users are commenting that they have the exact same problem.

https://stackoverflow.com/questions/42772599

I posted this exact issue to the EntityFramework6 Repository and they recommended that I come here looking for ideas.

[odinnix]

Yep, seeing this as well. Super annoying. It seems to be related to IIS Express - is there a git for that?

[danmosemsft]

@DamianEdwards where do IIS Express issues go?

[DamianEdwards]

Best place is likely at https://github.com/aspnet/AspNetCoreModule
IIS Express itself isn't open source so doesn't have its own repo, but this will raise it with the right people and is related.

[shirhatti]

@danmosemsft Why do you suspect this is an IIS Express issue? We use the ASP.NET Core Module to host dotnet.exe out of process from the IIS Express process. While we can certainly look into it, there's nothing about a FatalExecutionEngineError in a child process to indicate IIS Express is at fault.

[odinnix]

@shirhatti I actually think this is a Visual Studio debugger issue.

We have been able to work around the issue in two ways:

• dotnet run from the CLI
• Start IIS Express without the debugger attached, and then attach the debugger.

Any thoughts on a better place to post this?

[gkhanna79]

@gregg-miskelly Which is the debugger repo where this issue can be moved to?

[gregg-miskelly]

@gkhanna79 This issue should likely stay here as I don't really remember a 'FatalExecutionEngineError' while debugging issue ever actually being a problem with the debugger.

[gregg-miskelly]

@noahfalk does the CLR have a nob that @dcarl1 can use to get a dump of the problem?

[gkhanna79]

@adam8797 @shirhatti Do you have a repro or a crash dump for the FatalExecutionEngineError?

[adam8797]

@gkhanna79 I can't share the repo as its private to my company, but I can answer questions about it if you have any.

If you've got instructions on how to produce a dump file, I can do so tomorrow.

[gkhanna79]

It will be great if you can try to create a reduced repro. That said, can you use https://blogs.msdn.microsoft.com/cobold/2010/03/01/collecting-crash-dumps/ to see if you can generate the crash dumps? Do note that the crash dump could contain name of your app, filenames and paths, etc, if you are concerned about PII.

[adam8797]

Sounds fine to me. I'll see what I can throw together.

[odinnix]

Likewise. I'm doing some work this evening and will switch back to VS running to see if I can get a crash dump.

[gregg-miskelly]

Unfortunately, none of the techniques in that post will work as your issue only repros while the app is already being debugged by another debugger. Noah may have some suggestions...

[noahfalk]

ProcDump and Windbg can both handle the job, we just need to use them slightly differently than that page describes. Instructions for ProcDump:

1) Download the tool from https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
2) Unzip/unblock the exe and lets say you put it in C:\temp
3) Trigger the FatalExecutionEngineException popup and then without doing anything further in the VS UI switch to a console window
4) Run: C:\temp\procdump[64].exe -ma [process id]
If your app that crashed is a 64 bit app then you want to run procdump64, otherwise use procdump. If you aren't sure what bitness the app is an easy way I check from VS is open the debugger modules window and look in the address column. If the addresses are reported with 8 hex digits => 32 bit, 16 hex digits => 64 bit

It should look like this:

c:\Users\Noah\Desktop>procdump.exe -ma 8792

ProcDump v8.2 - Sysinternals process dump utility
Copyright (C) 2009-2016 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

[05:47:11] Dump 1 initiated: c:\Users\Noah\Desktop\App1.exe_170421_054711.dmp
[05:47:11] Dump 1 writing: Estimated dump file size is 165 MB.
[05:47:11] Dump 1 complete: 165 MB written in 0.3 seconds
[05:47:11] Dump count reached.

Then just send us that dump and we can take a peak under the hood.

[adam8797]

I've got the dump files, log files, and an example project.

https://github.com/adam8797/coreclr_Issue_10717

But it was very strange. I was not able to File -> New Project and make it crash. Instead I had to take my application, and strip it down to basically nothing.

I think it has something to do with my model classes. If I removed all of my models, I could not make it crash. Plus, I noticed that when it crashes, it always does so right before this line in the log file.

'ManagerReports.exe' (CLR v4.0.30319: ManagerReports.exe): Loaded 'EntityFrameworkDynamicProxies-ManagerReports'. 

If I see this line printed, I know its going to load correctly. Otherwise, it crashes right before this.

[odinnix]

@adam8797 I wonder if this is an issue with ComplexType or an abstract base class. Those are the two similarities between the models in this github project and what we use. Maybe take them off and see if you can repo?

[gregg-miskelly]

If someone from the CLR team wants to take a look, here are some pointers.

The dump to look at is the ManagerReports file. Look at thread 18 in windbg (or 19 in VS).

Here is the exception record from the dump:

-       pExceptionInfo->ExceptionRecord,x   0x06ccc3a4 {ExceptionCode=0xc0000005 ExceptionFlags=0x00000000 ExceptionRecord=0x00000000 <NULL> ...}   _EXCEPTION_RECORD *
        ExceptionCode   0xc0000005  long
        ExceptionFlags  0x00000000  unsigned long
+       ExceptionRecord 0x00000000 <NULL>   _EXCEPTION_RECORD *
        ExceptionAddress    0x0f501a09 {clr.dll!CoTemplate_qqhxpzqp(unsigned __int64), Line 8172}   void *
        NumberParameters    0x00000002  unsigned long
+       ExceptionInformation    0x06ccc3b8 {0x00000000, 0x059f4000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, ...}    unsigned long[0x0000000f]

ContextRecord : 0x06ccc3f4 {ContextFlags=65663 Dr0=0 Dr1=0 ...} _CONTEXT *

Stack:

    clr.dll!CoTemplate_qqhxpzqp(unsigned __int64) Line 8172 C++
    clr.dll!SVR::gc_heap::fire_etw_allocation_event(unsigned int allocation_amount, int gen_number, unsigned char * object_address) Line 653    C++
    clr.dll!SVR::gc_heap::try_allocate_more_space(alloc_context * acontext, unsigned int size, int gen_number) Line 12919   C++
    clr.dll!SVR::gc_heap::allocate_more_space(alloc_context * acontext, unsigned int size, int alloc_generation_number) Line 13179  C++
    clr.dll!SVR::GCHeap::Alloc(alloc_context * acontext, unsigned int size, unsigned int flags) Line 34616  C++
    clr.dll!Alloc(unsigned int size, int bFinalize, int bContainsPointers) Line 106 C++
    clr.dll!AllocateObject(MethodTable * pMT, bool fHandleCom) Line 993 C++
    clr.dll!JIT_New(CORINFO_CLASS_STRUCT_ * typeHnd_) Line 2814 C++

[jkotas]

This is use-after-free bug fixed in https://github.com/dotnet/coreclr/pull/6571#r73166763.

It will only crash with ETW managed memory tracing turned on, and program containing (generic) types with very long names. @gregg-miskelly Does the debugger turn on the ETW managed memory tracing by default?

[jkotas]

This crash is actually on .NET Framework, not on .NET Core.

@swgillespie @adityamandaleeka Could you please open an internal TFS bug to get the fix for this ported to .NET Framework vNext?

[gregg-miskelly]

@jkotas we turn on some ETW events, but I am not aware of anything in Visual Studio turning on anything related to managed memory. I can ask the profiler team if they know of anything.

[ChuckBryan]

I am having the same issue, similar stack. Back in March, we changed from "Any CPU" to "x64" and we didn't experience the Fatal Exception. Yesterday, we switched to "x86" and the problem came back. I switched from running IIS Express to running the Exe and haven't had the problem. If you need any additional crash dumps, I most likely could produce one.

[jkotas]

This is crash in full .NET Framework. I have opened .NET Framework on it (DevDiv TFS bug 425084). It will be fixed one in future .NET Framework updates.

The crash is triggered by turned on ETW GC memory tracing. To workaround it, you can try turning off ETW GC memory tracing.

[adam8797]

@jkotas Thanks for your help with this.

Any documentation on how to turn off ETW GC memory tracing in VS 2017?

[gregg-miskelly]

@adam8797 you can try this:

1. Open Tools->Options
2. Type in 'Diagnostics Tools' in the search bar
3. Uncheck 'Enable Diagnostics Tools while debugging'

[DavePilks]

I have tried this workaround but it didnt work for me. Any other suggestions?

[DavePilks]

We managed to work around this by turning off JavaScript Debugging.

[gregg-miskelly]

No idea how having JavaScript debugging on could possibly effect this, but I am glad you have something.

[DavePilks]

@gregg-miskelly it was the Javascript debugging in VS that I had to turn off. I assume that it is the same issue that other debuggers were experiencing.

[Trapulo]

Same problem. ASP.NET Core 2.0 target to NET 4.6.2 with EF 6.1.3

disabling diag tool while debugging seems a workaround

[chillNZ]

Turning off ETW GC memory tracing didn't work for me, but taking the advice of the people that said it was an x86 vs x64 issue I forced my IIS Express to run in 64 bit mode and it seems to have worked around the issue for me. To do this:
Start Visual Studio and click on Tools -> Options.
From the tree: Projects and Solutions, then Web Projects.
And make sure that Use the 64 bit version of IIS Express for web sites and projects is checked (press OK)

[MBurtsev]

I have this problem in source line
var str = new FileStream(fullPath, FileMode.OpenOrCreate, fa, FileShare.ReadWrite, 1024);

.Net 4.6.1 and 4.7.2

[OpenSpacesAndPlaces]

Any fix to this?

Seeing the problem in 4.7.2 using Microsoft.AspNetCore.SignalR 1.0.3 and Microsoft.EntityFrameworkCore 2.1.4.

IIS express is running in 64bit
Diagnostics off
Javascript debugging off.

It usually dies at following, but it can be random:
GlobalConfiguration.Configure in Global.Asax
or new HubConnection().StartAsync() e in Global.Asax

Specifically its an issue running without a project file (webforms - open website).

I'd say just use project file, but then it effectively creates the overlying problem again, which is being forced to stop/start the app vs. dynamic recompile working correctly.

Which if you are doing heavy .cs file editing, it is very frustrating to constantly start/stop.

[OpenSpacesAndPlaces]

@jkotas ^

[gregg-miskelly]

@OpenSpacesAndPlaces 4.7.2 would have a fix for this issue. So I doubt that the problem you are running into is related.

[OpenSpacesAndPlaces]

@gregg-miskelly

Managed Debugging Assistant 'FatalExecutionEngineError' 
  Message=Managed Debugging Assistant 'FatalExecutionEngineError' : 'The runtime has encountered a fatal error. The address of the error was at 0x72b0226e, on thread 0x208c. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.'

I've yet to run into anything that fixes the problem for me.

[gregg-miskelly]

A FatalExecutionEngineError means that something in your process has corrupted the runtime. I don't know what that thing is, but it is very unlikely to be related to this bug. If you can reproduce the problem with a native debugger attached, you could stop on that access violation (if you use VS, it will stop on native access violation exceptions by default) and maybe that will give you some ideas as to what is going wrong. If that address is within clr.dll you could try reporting this issue either through a new github issue here, or from Help->Send Feedback->Report a problem in VS.

If you can't reproduce the issue with a native debugger attached, that will make it much harder for the CLR team to understand. You could still save a .dmp file when stopped at the Managed Debugging Assistant so that you could find out something. If you debug the .dmp with with the interop debugging you could at the very least you open up the disassembly window and type in the address from the MDA text to see what function the exception is in.

[OpenSpacesAndPlaces]

The native error is:
Exception thrown at 0x72B0226E (clr.dll) in iisexpress.exe: 0xC0000005: Access violation reading location 0x00000040.

The dmp error is:

iisexpress.exe : C:\Program Files (x86)\IIS Express\iisexpress.exe
0xC0000005
The thread tried to read from or write to a virtual address for which it does not have the appropriate access.

[gregg-miskelly]

I am not on the CLR team. But I would suggest opening that .dmp in VS so you can grab a call stack and openning a new issue with that (unless you already see an issue with a similar stack). I would hang onto the .dmp as someone on the CLR team may contact you to obtain it.

[OpenSpacesAndPlaces]

Oddly this just stopped happening for my Team and I as of the latest Windows update. (in the past day).
So I guess speak of the devil and it gets patched?