Ef6: Reduce start up time by loading finished code first models from a persistent cache

[divega]

(Copied description from PR https://github.com/aspnet/EntityFramework6/pull/40)

Ported from CodePlex Pull Request 8468
Fix for CodePlex Work Items #1876 and #2631
Reduce start up time by loading finished Code First models from a persistent cache
Incorporates DbModelStore persistent cache to reduce Code First Startup Time

With these changes, first AppDomain calls to context.Database.Initialize for a model with just over 600 models and a null initializer dropped from 12-14 seconds to about 1.9 seconds after the edmx was written, saving 10-12 seconds on initialization. The first call to write the edmx still ran in 12-14 seconds (no noticeable delay added).

Includes code from the patch by emilcicos attached to work item #1876, with modifications to fix known issues.

Code changed after applying the patch:

• DefaultDbModelStore invalidates (deletes) the persistent store based on comparing context dll and edmx file last updated timestamps.
• EdmxWriter was updated to read from the DbModelStore when available. This fixed a null reference when using DropCreateAlways initializer with a DbModelStore (compiledModel.CachedModelBuilder is null when the model is loaded from the store). Added function DbModelStore.TryGetEdmx to allow the EdmxWriter to read the edmx directly from storage.
• LazyInternalContext.ModelMatches changes in the patch were removed as they were unnecessary after the EdmxWriter fix.
• Added unit and functional test coverage for code changes including the patch code. DefaultDbModelStoreTests functional test covers the end-to-end functionality of using a DbModelStore.

[divega]

Fixed with @j-Edge contribution at https://github.com/aspnet/EntityFramework6/pull/40.

[tlk]

What is the recommended way to get notified when this is released (6.2)?

[ErikEJ]

@tlk Watch the repo ?

[tlk]

@ErikEJ I would prefer to receive announcements only instead of project-wide chit-chat like this, but I'll watch the repo.

Looking very much forward to the improvement on startup time. Thanks!

[ErikEJ]

@tlk Understand, this repo however is very quiet. Alternatively, subscribe to the .NET blog RSS feed.

[NicoJuicy]

@divega How would i set this on DbMigrationsConfiguration(Of DataContext)?

This is in DbConfiguration, but not in DbMigrationsConfiguration.

this.SetModelStore(new DefaultDbModelStore(Directory.GetCurrentDirectory()));

[michalczerwinski]

I have a question: is the DefaultDbModelStore suppose to be set as default in 6.2?

[divega]

@NicoJuicy this can only be set in a DbConfiguration

@michalczerwinski it is not set by default, you have to opt in explicitly. That is something we could consider for a future release if the serialization as well as the invalidation mechanism proves to be very reliable.

[TimRowe]

This is as improvement of the code first mode , why you say it has to do with edmx file? edmx file isn't with the db first mode?

[divega]

@TimRowe the improvement serializes the computed code first model to an artifact that uses the edmx file format. I.e. the edmx becomes an implementation detail of code first when you are using this option.

[brux88]

Hi,but if i use code first migrations , how i can do use persistent caching? My database configuration use dbmigrationsconfiguration and not use dbconfiguration

[abdelrady]

FYI .. this item has been released! EF6.2.0 is out!
https://blogs.msdn.microsoft.com/dotnet/2017/10/26/entity-framework-6-2-runtime-released/

[SylwesterZarebski]

@divega commented on 9 lip 2017, 09:19 CEST:

@NicoJuicy this can only be set in a DbConfiguration

It means that is not possible to use persistent caching with DbMigrationsConfiguration?

[brux88]

@SylwesterZarebski you can use both

[SylwesterZarebski]

Could You share some code? Now i have:

Database.SetInitializer(new Migrations.CheckAndMigrateDatabaseToLatestVersion<AppContext, Migrations.Configuration>());

where CheckAndMigrateDatabaseToLatestVersion is:

internal sealed class CheckAndMigrateDatabaseToLatestVersion<TContext, TMigrationsConfiguration> : IDatabaseInitializer<TContext>
        where TContext : DbContext
        where TMigrationsConfiguration : DbMigrationsConfiguration<TContext>, new()

How can i add persistant caching?

[brux88]

sure.
I have two configuration file for my context (SednaContext).

this configuration for migrations:

```
public sealed class Configuration : DbMigrationsConfiguration
{
public Configuration()
{
AutomaticMigrationsEnabled = false;
// AutomaticMigrationDataLossAllowed = true;
MigrationsDirectory = @"Migrations_SednaContext";
}

    protected override void Seed(UnitOfWork.SednaContext context)
    {

    }

}

and this configuration for chaching:

public sealed class ConfigurationCaching : DbConfiguration
{
public ConfigurationCaching(): base()
{
var pathEDMX = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), @"Sedna\EDMX");

        if (!Directory.Exists(pathEDMX))
        {
            DirectorySecurity securityRules = new DirectorySecurity();
            securityRules.AddAccessRule(new FileSystemAccessRule(
                                        new SecurityIdentifier(WellKnownSidType.WorldSid, null),
                                        FileSystemRights.FullControl, InheritanceFlags.ObjectInherit |
                                        InheritanceFlags.ContainerInherit, PropagationFlags.NoPropagateInherit, AccessControlType.Allow));
            Directory.CreateDirectory(pathEDMX, securityRules);
        }
        this.SetModelStore(new DefaultDbModelStore(pathEDMX/*Directory.GetCurrentDirectory()*/));
    }

}

then you set attribute on your DbContext file:

[DbConfigurationType(typeof(ConfigurationCaching))]
public class SednaContext : DbContext
{
...
```

[SylwesterZarebski]

Thanks, i'll try to use it that way.

[srudin]

This is not working for me.

What I did is very simple and I think exactly what has been suggested above:

I saved the model using
EdmxWriter.WriteEdmx(context, writer);

In my DbConfiguration implementation I set
SetModelStore(new DbModelStore());

DbModelStore is my implementation of DefaultDbModelStore where I overwrite GetFilePath().

When making the first call to EntitySet I get
InvalidOperationException: The entity type EdmMetadata is not part of the model for the current context.
System.Data.Entity.Internal.InternalContext.UpdateEntitySetMappingsForType(Type entityType) +273
System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +31
System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77 System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29
System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75 System.Data.Entity.Internal.EdmMetadataRepository.QueryForModelHash(Func2 createContext) +159
System.Data.Entity.Internal.ModelCompatibilityChecker.CompatibleWithModel(InternalContext internalContext, ModelHashCalculator modelHashCalculator, Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +120
System.Data.Entity.Internal.InternalContext.CompatibleWithModel(Boolean throwIfNoMetadata, DatabaseExistenceState existenceState) +77
System.Data.Entity.CreateDatabaseIfNotExists1.InitializeDatabase(TContext context) +155 System.Data.Entity.Internal.InternalContext.PerformInitializationAction(Action action) +71 System.Data.Entity.Internal.InternalContext.PerformDatabaseInitialization() +482 System.Data.Entity.Internal.RetryAction1.PerformAction(TInput input) +174
System.Data.Entity.Internal.LazyInternalContext.InitializeDatabaseAction(Action1 action) +273 System.Data.Entity.Internal.InternalContext.GetEntitySetAndBaseTypeForType(Type entityType) +20 System.Data.Entity.Internal.Linq.InternalSet1.Initialize() +77
System.Data.Entity.Internal.Linq.InternalSet1.AsNoTracking() +29 System.Data.Entity.Infrastructure.DbQuery1.AsNoTracking() +75
[my call to EntitySet]

I have been able to get it to work when I create a DbCompiledModel instead:
var model = EdmxReader.Read(reader, "dbo");

I can then pass the model to the constructor of the context. However this is much more complicated mostly because of the stupid null check of the model in the constructor (!) so I have to make sure to call the right constructor overload depending on whether I already have a saved model or not.

A fix for the null check or an explanation why the DefaultDbModelStore is not working would be appreciated. I consider both to be bugs.

[srudin]

Created a new issue instead: https://github.com/aspnet/EntityFramework6/issues/404

[ivanovevgeny]

My project's start time is about 10 sec.
I use migrations.

I added this model caching feature

```c#
public class ProductsDbConfiguration : DbConfiguration
{
public ProductsDbConfiguration() : base()
{
this.SetModelStore(new DefaultDbModelStore(AppDomain.CurrentDomain.BaseDirectory));
}
}

[DbConfigurationType(typeof(ProductsDbConfiguration))]
public class ProductsContext : DbContext
{...}

Edmx file created succesffully on first run. 
Than i changed some service, model is located in another project in solution and has not been changed. I had not see any improvements on loading time - 10 sec.

**What i'm doing wrong?**

Migration init code:
```c#
// Global.asax.cs --------
Database.SetInitializer(new MigrateDatabaseToLatestVersion<ProductsContext, Products.Infrastructure.Migrations.Configuration>());
// ---------

namespace Products.Infrastructure.Migrations
{
    using System.Data.Entity.Migrations;

    public sealed class Configuration : DbMigrationsConfiguration<Repositories.ProductsContext>
    {
        public Configuration()
        {
            AutomaticMigrationsEnabled = false;
            CommandTimeout = 60 * 5; 
        }
    }
}

[ajcvickers]

@ivanovevgeny It could be that Code First model creation is not the perf bottleneck. It might be worth reading EF perf white paper and checking to see if any of the guidance there can help.