Efcore: System.ExecutionEngineException when IIS application recycle

[Tasteful]

We are running EFCore 2.0.1 (also tested with 2.0.2) inside Full .Net IIS Application. Tested with Framework version 4.6.1 and 4.7.1 with the same problem.

When saving the web.config to restart the site (also on rebuild the solution) the System.ExecutionEngineException is thrown and a Just-In-Time debugger popup asks to debug the process. After the site have been termidated of the exception it will start again without problem. I cant find in the log the exception has occur before ´2018-03-31 12:42:45` and the application code is not changed either. No updates/security updates have been installed on the machine around this date either. Exception is thrown on atleast two differente computers and working on four other computers.

How can I find out what is causing this System.ExecutionEngineException and is that thrown from EFCore in some known scenarios?

PS> Get-HotFix | Sort-Object InstalledOn

Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
PC-STHLM-R... Update           KB4058702     NT AUTHORITY\SYSTEM  2018-01-05 00:00:00
PC-STHLM-R... Security Update  KB4056887     NT AUTHORITY\SYSTEM  2018-01-19 00:00:00
PC-STHLM-R... Update           KB4074608     NT AUTHORITY\SYSTEM  2018-02-05 00:00:00
PC-STHLM-R... Security Update  KB4074595     NT AUTHORITY\SYSTEM  2018-02-12 00:00:00
PC-STHLM-R... Update           KB4087256     NT AUTHORITY\SYSTEM  2018-03-01 00:00:00
PC-STHLM-R... Update           KB4090914     NT AUTHORITY\SYSTEM  2018-03-07 00:00:00
PC-STHLM-R... Security Update  KB4088785     NT AUTHORITY\SYSTEM  2018-03-16 00:00:00
PC-STHLM-R... Security Update  KB4088776     NT AUTHORITY\SYSTEM  2018-03-20 00:00:00

Exception information

Exception message: Exception of type 'System.ExecutionEngineException' was thrown.
Exception does not contains any stack trace. 

Memorydump provides with clrstack starting like this (memorydump can be sent on request).

00000067803785c0 00007ff95e12093c [PrestubMethodFrame: 00000067803785c0] Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.SharedTableConvention.Apply(Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder)
0000006780378880 00007ff908e5fe76 Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher+ImmediateConventionScope.OnModelBuilt(Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder)
00000067803788c0 00007ff9087ce75f Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(Microsoft.EntityFrameworkCore.DbContext, Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.IConventionSetBuilder, Microsoft.EntityFrameworkCore.Infrastructure.IModelValidator)
0000006780378920 00007ff95c831774 System.Collections.Concurrent.ConcurrentDictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].GetOrAdd(System.__Canon, System.Func`2<System.__Canon,System.__Canon>)
0000006780378980 00007ff9087d4bd1 Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()
00000067803789e0 00007ff9087d4db2 Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()

Steps to reproduce

Not aware of reason so not able to create simple reproduction steps. Project is an commercial e-commerce application so I can't share the code on github.

Further technical details

EF Core version: 2.0.1
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows 10 all updates installed
IDE: Visual Studio 2017 15.6.4

[divega]

@Tasteful could you share the project privately? E.g. on email so that we can diagnose the issue?

Also, in case this is an issue with the conventions using too much memory, could you try with our nightly builds of EF Core 2.1? We have fixed a few bugs around that area.

[Tasteful]

@divega I have invited you to an one-drive folder that contains the running code and both a mini and full memory dump.

I actually think it is some special conditions that depend on some other program or hardware problem but have no clue how to find the responsible part.

I have tried the following parts and can only get the exception on my computer, using 10 of 16 GB memory.

• Trying with the EFCore package 2.1.0-preview3-32121 from myget - still exceptions
• Copied the code into a Windows Server 2012r2 - Running without problem
• Created a new Windows 10 VM from VS Enteprise Azure image (VS 15.0.0.0), installed IIS, SQL 2017 Express, Updated to VS 15.6.4 - Running without problem
• Restarted machine, and clearing all temp files (also asp.net temporary files in microsoft.net framework folder)
• Update Visual Studio to 15.6.5 - still exception
• Copied the artifacts from build server for the same branch to local machine, the branch is working on test server - still exceptions localy

[divega]

@AndriySvyryd I believe when we originally triaged this, you had a couple of ideas of what could cause it (running out of memory?).

Unless you think you can help, it might be better to move this to CoreCLR or CoreFx, as it is unlikely that the root cause for this type of exception is something EF Core is doing.

You can ask @Tasteful to share the repro with you.

[Tasteful]

@AndriySvyryd if you send me your email I can share the onedrive folder with you, you have my mail in the profile.

[divega]

@Tasteful Sorry for the long wait. I discussed this with @AndriySvyryd and we actually don't have a good clue of what can cause this. This type of exception usually happens when something goes wrong at a very low level. E.g. when invalid IL is executed.

Since in EF Core is pretty high level and we don't do our own IL code generation, the root cause of the issue must be somewhere else. E.g. in the .NET Core Runtime, the C# compiler or in the LINQ expression compiler that is part of CoreFX.

I am going to move the issue to one of the reposistories, hoping that someone with experience debugging this kind of issues will be able to take your repro and identify the root cause.

[divega]

This issue was moved to dotnet/coreclr#17864

[NickCraver]

I'm bringing the issue back here as my findings thus far point to an EF/DI issue (and both are equally closed):

Alrighty I finally got a proper stack here. It's being thrown at:

Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Storage.RelationalTypeMappingSource.FindMapping(System.Reflection.MemberInfo member) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.PropertyDiscoveryConvention.IsCandidatePrimitiveProperty(System.Reflection.PropertyInfo propertyInfo) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.PropertyDiscoveryConvention.Apply(Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder entityTypeBuilder = {Microsoft.EntityFrameworkCore.Metadata.Internal.EntityType}) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ImmediateConventionScope.OnEntityTypeAdded(Microsoft.EntityFrameworkCore.Metadata.Internal.InternalEntityTypeBuilder entityTypeBuilder = {Microsoft.EntityFrameworkCore.Metadata.Internal.EntityType})   Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.RunVisitor.VisitOnEntityTypeAdded(Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.OnEntityTypeAddedNode node)   Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionVisitor.VisitConventionScope(Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionScope node = {Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionScope})   Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Run()    Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.ConventionDispatcher.ConventionBatch.Dispose()    Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(Microsoft.EntityFrameworkCore.Metadata.Internal.TypeIdentity type, Microsoft.EntityFrameworkCore.Metadata.Internal.ConfigurationSource configurationSource, bool throwOnQuery)    Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Metadata.Internal.InternalModelBuilder.Entity(System.Type type, Microsoft.EntityFrameworkCore.Metadata.Internal.ConfigurationSource configurationSource, bool throwOnQuery) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.ModelBuilder.Entity(System.Type type)   Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Infrastructure.ModelCustomizer.FindSets(Microsoft.EntityFrameworkCore.ModelBuilder modelBuilder = {Microsoft.EntityFrameworkCore.ModelBuilder}, Microsoft.EntityFrameworkCore.DbContext context)    Unknown
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.FindSets(Microsoft.EntityFrameworkCore.ModelBuilder modelBuilder = {Microsoft.EntityFrameworkCore.ModelBuilder}, Microsoft.EntityFrameworkCore.DbContext context = {StackOverflow.Models.SitesDBContext})   Unknown
Microsoft.EntityFrameworkCore.Relational.dll!Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelCustomizer.Customize(Microsoft.EntityFrameworkCore.ModelBuilder modelBuilder = {Microsoft.EntityFrameworkCore.ModelBuilder}, Microsoft.EntityFrameworkCore.DbContext context = {StackOverflow.Models.SitesDBContext})  Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.CreateModel(Microsoft.EntityFrameworkCore.DbContext context, Microsoft.EntityFrameworkCore.Metadata.Conventions.Internal.IConventionSetBuilder conventionSetBuilder, Microsoft.EntityFrameworkCore.Infrastructure.IModelValidator validator = {Microsoft.EntityFrameworkCore.Internal.SqlServerModelValidator})  Unknown
mscorlib.dll!System.Lazy<Microsoft.EntityFrameworkCore.Metadata.IModel>.CreateValue()   Unknown
mscorlib.dll!System.Lazy<Microsoft.EntityFrameworkCore.Metadata.IModel>.LazyInitValue() Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel()    Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Internal.DbContextServices.Model.get()  Unknown
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite scopedCallSite = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite}, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope scope = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope})  Unknown
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite constructorCallSite = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite}, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope scope = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope})  Unknown
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScoped(Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite scopedCallSite = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ScopedCallSite}, Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope scope = {Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope})  Unknown
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.RealizeService.AnonymousMethod__0(Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope scope) Unknown
Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(System.IServiceProvider provider, System.Type serviceType = {Name = "IDbContextDependencies" FullName = "Microsoft.EntityFrameworkCore.Internal.IDbContextDependencies"})    Unknown
Microsoft.Extensions.DependencyInjection.Abstractions.dll!Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService<Microsoft.EntityFrameworkCore.Internal.IDbContextDependencies>(System.IServiceProvider provider) Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.DbContext.DbContextDependencies.get()   Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.DbContext.InternalServiceProvider.get() Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Internal.InternalAccessorExtensions.GetService<Microsoft.EntityFrameworkCore.Storage.IDbContextTransactionManager>(Microsoft.EntityFrameworkCore.Infrastructure.IInfrastructure<System.IServiceProvider> accessor)  Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.TransactionManager.get()  Unknown
Microsoft.EntityFrameworkCore.dll!Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.CurrentTransaction.get()  Unknown
>   StackOverflow.dll!StackOverflow.Models.StackDbContext.Transaction.get() Line 22 C#
StackOverflow.dll!StackOverflow.Models.StackDbContext.Query<StackOverflow.Models.Site>(string sql = "select *, State as StateRaw from Sites where SiteTypeId <> 14", object param = null, int? commandTimeout = null, System.Data.IDbTransaction transaction = null, string fromFile = "C:\\code\\stackoverflow\\StackOverflow\\Models\\Site.cs", int onLine = 873, string comment = null) Line 250 C#
StackOverflow.dll!StackOverflow.Models.Site.ReloadSitesCache() Line 873 C#
StackOverflow.dll!StackOverflow.Models.Site.SitesPrivate.get() Line 785 C#
StackOverflow.dll!StackOverflow.Models.Site.GetSiteByBaseHostAddress(string baseHostAddress = "local.so.com", bool demand = false) Line 922 C#
StackOverflow.dll!StackOverflow.ExplicitSiteContextProvider.ExplicitSiteContextProvider(string baseHostAddress = "local.so.com", StackOverflow.Models.Site site = null) Line 91 C#
StackOverflow.dll!StackOverflow.Current.Current() Line 924  C#
[Native to Managed Transition]  
[Managed to Native Transition]  
StackOverflow.dll!StackOverflow.Models.VerificationHashHelper.GetKey() Line 896 C#
StackOverflow.dll!StackOverflow.Models.VerificationHashHelper.VerificationHashHelper() Line 50  C#
[Native to Managed Transition]  
[Managed to Native Transition]  
StackOverflow.dll!StackOverflow.GlobalApplication.Application_Start(object sender = {System.Web.HttpApplicationFactory}, System.EventArgs e = {System.EventArgs}) Line 38   C#
[Native to Managed Transition]  
[Managed to Native Transition]  
System.Web.dll!System.Web.HttpApplication.InvokeMethodWithAssert(System.Reflection.MethodInfo method, int paramCount, object eventSource, System.EventArgs eventArgs)   Unknown
System.Web.dll!System.Web.HttpApplication.ProcessSpecialRequest(System.Web.HttpContext context = {System.Web.HttpContext}, System.Reflection.MethodInfo method, int paramCount, object eventSource, System.EventArgs eventArgs, System.Web.SessionState.HttpSessionState session)   Unknown
System.Web.dll!System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(System.Web.HttpContext context = {System.Web.HttpContext}, System.Web.HttpApplication app)   Unknown
System.Web.dll!System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(System.IntPtr appContext = 0x000001eacbb9ad20, System.Web.HttpContext context = {System.Web.HttpContext}, System.Reflection.MethodInfo[] handlers)  Unknown
System.Web.dll!System.Web.HttpApplication.InitSpecial(System.Web.HttpApplicationState state, System.Reflection.MethodInfo[] handlers, System.IntPtr appContext = 0x000001eacbb9ad20, System.Web.HttpContext context)    Unknown
System.Web.dll!System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(System.IntPtr appContext, System.Web.HttpContext context)    Unknown
System.Web.dll!System.Web.Hosting.PipelineRuntime.InitializeApplication(System.IntPtr appContext = 0x000001eacbb9ad20)  Unknown
[AppDomain Transition]  

I'm going to dig into that source next, but I have a memory dump of the above (1.7GB) and can pop it on a Google drive for NDA share.

Note the call it's registering is on the abstract:
```c#
protected abstract RelationalTypeMapping FindMapping(in RelationalTypeMappingInfo mappingInfo);

Which should be calling into [the `SqlServerTypeMappingSource` implementation](https://github.com/aspnet/EntityFrameworkCore/blob/2.1.1/src/EFCore.SqlServer/Storage/Internal/SqlServerTypeMappingSource.cs#L227) (but maybe it isn't registered yet? since we're in `Application_Start`?):
```c#
protected override RelationalTypeMapping FindMapping(in RelationalTypeMappingInfo mappingInfo)
    => FindRawMapping(mappingInfo)?.Clone(mappingInfo);

So I'm not sure it's a CLR issue. It feels at the moment like a DI issue returning an abstract class early in the application lifetime. We don't use DI at all for anything else, and we're on ASP.NET (not Core), so I'd appreciate someone more familiar taking a sanity look here.

/cc @divega

[divega]

@NickCraver, definitively if you can put it the crash dump in a Google drive.

@ajcvickers, @davidfowl, @Eilon may have some insights on the theory that this could be a problem with using DI early from inside Application_Start().

[divega]

This is an issue in .NET Framework that is being tracked to be fixed in a future .NET Framework release:

If

1. An assembly's IL metadata contains one or more AssemblyRefs that are referenced only by type forwarders, and

2. The assembly is NGen'ed

then these only-for-forwarders AssemblyRefs are currently omitted from the binding closure computed when determining whether the native image can be loaded domain-neutral. See the trailing section below for details.

This leads to many situations where crashed occur because the native image ends up being illegally loaded into the shared domain even though it has domain-bound dependencies.

As described below, a point of particular concern is the impact that this bug can have in scenarios netstandard facade assembly.

Workarounds

1. Make sure the application targets .NET Framework 4.7.1 or higher: the bug is triggered by app-local deployed netstandard assemblies. The app-local deployed netstandard assemblies are needed for apps targeting .NET Framework 4.7 or earlier (because the netstandard assemblies are not in-box). Change the app to target .NET Framework 4.7.1 and this will avoid the need for app-local deployed netstandard assemblies and binding redirects.

2. On 64-bit runtime you can disable loading of the native image using RyuJIT flags described in https://github.com/Microsoft/dotnet/blob/master/Documentation/testing-with-ryujit.md

   Specifically, there is a DisableNativeImageLoadList configuration element which can be used to achieve the same effect as the machine.config change via the registry or an environment variable. For example, the following environment variable setting has the same effect as the machine.config elements:

3. On both 32-bit and 64-bit runtimes you can achieve similar effects making sure you have these entries in machine.config (in either the framework or frmework64 folder, but not in the application’s config file):

   <runtime>
    <disableNativeImageLoad>
       <assemblyIdentity name="netstandard" />
     </disableNativeImageLoad>
   </runtime>