Efcore: .NET Native: Slow performance in change tracker

[Shayon]

I am building a UWP app with EF Core and am running into a massive performance issue when running in release with the .NET Native Tool Chain. Release is at least 4x slower than debug, but sometimes gets up to 10x slower. For example, in debug when I perform a specific operation I get roughly 10 seconds for it to complete, that SAME operation takes ~40 seconds when running in release. Using diagnostic sessions I have been able to determine that the slowdown is almost exclusively due to ChangeTracker.Entries and ChangeTracker.DetectChanges. See Screenshot:

I'd love to get to the bottom of this and hopefully find a temporary work-around while also providing as much information as necessary to the team to get this fixed for good.

Steps to reproduce

REPO SET UP TO REPRODUCE THE ISSUE: https://github.com/Shayon/uwp-sample/tree/ef_perf_issue

In the application I am building I query an API to get a graph of items and then use a pattern of recursive calls to what I have dubbed Importers to load the graph into the database. Each Importer takes a list of WebEntites and adds or updates DB entities based on whether or not it can find an existing db entity from the ServerId which comes down from the API. On a DB entity, a ServerId is an indexed & unique column. A context is created and passed into my importer chain, and when the importer chain completes, I call SaveChanges(). My Importers look something like this:

```c#
public static class Importer
where TDb : BaseModel
where TApi : IHasServerId
{
public static IList Import(
DbContext context,
IList webEntities,
int scopeId,
Action copyFields)
{
var createdAndUpdatedDbEntities = new List();

    if (webEntities == null || !webEntities.Any()) { return createdAndUpdatedDbEntities; }

    var dbEntitiesToDelete =
        new HashSet<TDb>(
            context.Set<TDb>().Where(entity => entity.GetScopeId() == scopeId));

    foreach (var webEntity in webEntities)
    {
        var dbEntity = Import(context, webEntity, copyFields);
        createdAndUpdatedDbEntities.Add(dbEntity);
        dbEntitiesToDelete.Remove(dbEntity);
    }

    foreach (var priorDbEntity in dbEntitiesToDelete) {
        context.Entry(priorDbEntity).State = EntityState.Deleted;
    }

    return createdAndUpdatedDbEntities;
}

public static TDb Import(
    DbContext context,
    TApi webEntity,
    Action<TDb, TApi> copyFields)
{
    if (webEntity == null) { return null; }

    var dbEntity =
        context
            .ChangeTracker
            .Entries<TDb>()
            .Select(entry => entry.Entity)
            .FirstOrDefault(
                entity => entity.ServerId == webEntity.ServerId) ??
        context
            .Set<TDb>()
            .FirstOrDefault(
                entity => entity.ServerId == webEntity.ServerId);

    if (dbEntity == null)
    {
        dbEntity = Activator.CreateInstance<TDb>();
        dbEntity.ServerId = webEntity.ServerId;
        context.Entry(dbEntity).State = EntityState.Added;
    }

    copyFields(dbEntity, webEntity);
    context.ChangeTracker.DetectChanges();
    return dbEntity;
}

}

It may look like there is a lot going on here, but it is actually fairly simple.  Each Importer will get a:
 * `context`: database context created one level above that will handle importing the full object graph we got from the api.
 * `webEntities`: a list of our web models.
 * `scopeId`: tells us what scope we care about, essentially a FK.
 * `copyFields`: describes how to copy contents over from webModels to our DbModels.  This is where we have recursive calls to other Importers.

I suspected we may be over-abusing the `ChangeTracker.Entries` here: 
```c#
context
    .ChangeTracker
    .Entries<TDb>()
    .Select(entry => entry.Entity)
    .FirstOrDefault(
        entity => entity.ServerId == webEntity.ServerId

So I replaced it with my own version of a local change tracker (simply a global dictionary) and saw virtually no speed up.

I also tried running the application with VS2017 and updated my UWP package to 5.3.0 and saw no improvement.

I also tried updating to EF 1.1 and saw no improvement.

Further technical details

EF Core version: 1.0.0

Database Provider: Microsoft.EntityFrameworkCore.Sqlite

Operating system: Windows 10 1607

IDE: Visual Studio 2015

Microsoft.NETCore.UniversalWindowsPlatform Version 5.2.2

[allanrsmith]

@rowanmiller Is .net native perf with EF Core something we will just have to live with? It doesn't look like a light at the end of the tunnel. Is EF Core with UWP essentially not recommended?

[allanrsmith]

@divega @MichalStrehovsky help?

[rowanmiller]

Can you try upgrading Microsoft.NETCore.UniversalWindowsPlatform to 5.3.0?

[Shayon]

Hey @rowanmiller, we've been running some tests and we are able to make _some_ progress, take a look at some of our release-build diagnostic sessions:
Note: For all the pictures below, there is a first load, (importing data into an empty DB), and then an update load, (importing the exact same data as an update)

This is what we started out with on https://github.com/Shayon/uwp-sample/tree/ef_perf_issue

Pretty awfully slow

Then, this is that same branch with UWP 5.3

About 10 seconds of speed up, so at least it went it the right direction. But still horribly slow.

Then we tried adopting a different ChangeTracking Strategy, we added INPC to our models and used ChangingAndChangedNotifications, this is still back on UWP 5.2.2 (branch with these changes can be found here: https://github.com/Shayon/uwp-sample/tree/jg/1-1-with-inpc)

So a huge speed up by adopting the new change-tracking strategy.

Next, we tried the same changes as above, but also bumped the UWP nuget to 5.3

So much better than when we began 40 sec down to 5 sec on first load for the same set of sample data. But I still think 5 seconds to put less than 500 kb into a database is abysmally slow. It is also pretty concerning that we are still slower on Release than we are on Debug.

[rowanmiller]

We are going to test this on 1.1.1 and see if it is improved. If not, we'll hand of to .NET Native team to investigate.

[ajcvickers]

Note to self: follow up with .NET Native.

[AndriySvyryd]

@MichalStrehovsky This hasn't improved in 6.0.0 prerelease

[AndriySvyryd]

@Shayon In addition to using INPC add the attached configuration to your app RD.xml file. This will not be necessary with EF 2.1+

Also if the real model has many enum properties perf can be further improved by adding these lines to the RD.xml file:

<!-- Add these lines if you have entities with properties of an enum type, replacing 'MyCode.MyEnum' with the enum used -->
<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.ClrPropertyGetter{System.Object, MyCode.MyEnum}" Dynamic="Required Public" />
<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.ClrPropertySetter{System.Object, MyCode.MyEnum}" Dynamic="Required Public" />

<!-- Add these lines if you have entities with properties of a nullable enum type, replacing 'MyCode.MyEnum' with the enum used -->
<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.ClrPropertyGetter{System.Object, System.Nullable{MyCode.MyEnum}}" Dynamic="Required Public" />
<Type Name="Microsoft.EntityFrameworkCore.Metadata.Internal.NullableEnumClrPropertySetter{System.Object, System.Nullable{MyCode.MyEnum}, MyCode.MyEnum}" Dynamic="Required Public" />

Default.rd.xml.txt

[Shayon]

I'm unable to see any performance improvement. Applied the changes to Default.rd.xml and ran it in Release and saw no significant changes in speed. 😕

Do you have any performance numbers to show? Where should I expect performance improvement?

[AndriySvyryd]

@Shayon Using INPC these changes decrease the running time by 10% making it the same as Debug when running on the prerelease version of UWP 6.0.0 with prerelease version of EF Core 2.0.1.
Without INPC they net about x2 perf increase.
I've also opened https://github.com/aspnet/EntityFrameworkCore/issues/9422 to make this scenario faster in general.

[Shayon]

Okay, I was unable to test with prerelease version of UWP 6.0.0 and with prerelease version of EF Core 2.0.1, so hopefully they missing performance will appear when I am able to bump versions.

I'm glad you've opened a more general ticket as well as this is still a pretty serious concern for us. Thanks.