Efcore: Extremely slow build for asp.net core 3.1 apps

[rezathecoder]

I’m using visual studio community 2019 for developing asp.net core applications but i experience extremely slow builds that is really preventing me from doing my work. So here is the situation:

I was using version 16.7.7 of visual studio 2019 on windows 10 20H2 update and when i started build process it took about 10 MINUTES each time. So i updated to version 16.8.2 and things got worse.

Now it takes more than 30 MINUTES to build my project. During build VBCSCompiler process uses 100% of CPU power and more than 6GB of RAM and i’m unable to use my computer in the mean time. I run on SSD so it’s not a problem related to slow disk.

I use asp.net core 3.1.9 mvc and i’m working on a medium sized project with about 800 CS files and 400 razor views and a few .resx files.

The build configuration is set on Any CPU and there is no difference between debug and release modes.

Important Note:

When i make a change in views (.cshtml files) the build time is about 30 SECONDS which is normal and acceptable but if i change in CS files even only one line of code in a single file i have to wait 30 MINUTES to see the result.
I don’t think it’s related but i use Git for source control and NPM for managing client side libraries.

Please help me with this problem. Thanks!

[rezathecoder]

I did a little investigation myself and here is the result:
I created a new project and i installed any nuget packages that had in the main project. Then i started to add files one by one from main project to new project and measured build time to see where this slow build comes from. I added all files (.cs, .cshtml, ...) to new project and everything was normal with build time about 1 MINUTE.

But when i added 'Migrations' folder to new project i noticed that build got too slow. Then i found that every migration has a .Designer.cs file which is about 2.5MB in size so over 250MB of csharp files in Migrations folder!

So i deleted all migrations and started to add migrations one by one and i realized the build time is normal when i have a couple of migrations but when it pass certain number of migrations it starts to slow down.

It seems compiler is unable to manage codes when code size is larger than a threshold.

So any suggestion? Should i remove my migrations periodically (which is not a good solution because i wanna keep track of database changes that applied or needs to be applied on the deployment server) or compiler needs to be fixed?

[mkArtakMSFT]

Thanks for contacting us, @rezathecoder.

@ajcvickers I vaguely remember something similar to this. Does this ring any bells for you?

[mkArtakMSFT]

Maybe related to https://github.com/dotnet/efcore/issues/23426 ?

[mkArtakMSFT]

@rezathecoder can you try out this recommendation

[roji]

Then i found that every migration has a .Designer.cs file which is about 2.5MB in size so over 250MB of csharp files in Migrations folder!

@rezathecoder 2.5MB per migration is indeed problematic - may I ask what you're doing to reach these sizes? Are you by any chance seeding large data via migrations (#21065)?

[rezathecoder]

Then i found that every migration has a .Designer.cs file which is about 2.5MB in size so over 250MB of csharp files in Migrations folder!

@rezathecoder 2.5MB per migration is indeed problematic - may I ask what you're doing to reach these sizes? Are you by any chance seeding large data via migrations (#21065)?

Yes i am seeding my database but it's not binary data like the issue you mentioned and it's not very large.
I think the code related to seeding process should only exist in the first migration but why it is repeated in every single migration that i created after seeding?

[rezathecoder]

@rezathecoder can you try out this recommendation

Answering to the question asked in the comment you mentioned: Yes i experienced this slowness existed before in version 16.8 and 16.7 and earlier.
And about using daily build of dotnet SDK can you give me a reference in how to do it? I'm using asp.net core 3.1

[rezathecoder]

@mkArtakMSFT @roji @ajcvickers
As i mentioned before i think new migration .Designer.cs files should not contain the code for seeded data in previous migrations but i think there is a temporary solution and i wanted to know expert's opinion on that to see if there is any downside that i'm missing.

If we set the Build Action for old migrations that are already applied to the database (except the last one) to None they will be excluded from compilation process (and we get fast compilation) and it seem to generate correct SQL queries.
For example if we have 10 migrations and number 8 is the last applied migration on database, we could set Build Action for migrations number 1 to 7 to None and set the rest to C# Compiler then we can execute following command and get the correct script:
Script-Migration -From 8 -Context ApplicationContext

I also notice if i do this my [ProjectName].dll file size is 10 times smaller than before and i think compiled migration files has nothing to do with application functionality.

[roji]

As i mentioned before i think new migration .Designer.cs files should not contain the code for seeded data in previous migrations but i think there is a temporary solution and i wanted to know expert's opinion on that to see if there is any downside that i'm missing.

@rezathecoder the Designer file for each migration needs to contain this information in order for EF to function correctly. If you exclude old migration files from compilation, that obviously speeds up your compilation; however, you will no longer be able to apply those old migrations (e.g. when creating a new database), or generate migration SQL scripts. In the general case, users do need to have old migrations compiled into their projects, even if in your specific case compiling only the latest one may work.

The first thing to understand from your side, is what your model contains that makes each Designer file 2.5MB big; if this is seeded data (regardless of whether it's binary or not), this is discouraged as I wrote above (and as we mention in our docs). If seeding data what's causing your Designer files to be 2.5MB big, consider switching to custom initialization logic instead.

Finally, we are indeed currently investigating some reports that Visual Studio 16.8 has worsened the build times around migrations. One issue has already been located and fixed in VS 16.8.2, and I'm looking into another one which may explain the difference you're seeing between 16.7.x and 16.8.2. However, even your starting situation with VS 16.7.x, where your build takes 10 minutes, is abnormal, and is probably the result of big seeded data as I wrote above.

[rezathecoder]

@roji Thanks for your response. Yes the big size of migration files is related to seeding data and i will try to switch to custom initialization logic. I guess for now we could apply that temporary solution and exclude old migrations and if we needed to create new database just set the build action to C# Compiler again.
Thanks for your effort for making efcore better i'm waiting to hear good news from the team.

[rezathecoder]

@roji So i tried to switch to custom initialization logic but i think it's not suitable for me because i have a lot of base information that i need to seed into my database (about 10,000 records) that are rarely changed. According to code example in docs i need to check if the record exists or not then try to insert it and it can't be done for large data. I think efcore needs a new way to handle data seeding. I will post some ideas on that area in a new issue soon.

[ErikEJ]

@rezathecoder Where in the docs - you should be able to use MERGE with SQL Server.

[ajcvickers]

@roji How about this one?

[roji]

@rezathecoder coming back to this, just to make sure - your original post above mentions a build time regression from 10 minutes to 30 when upgrading from VS 16.7.x to 16.8.2. Can you please confirm this (https://github.com/dotnet/efcore/issues/23435#issuecomment-732140341 may suggest otherwise), and specifically that you're still experiencing a regression in 16.8.2 (as opposed to 16.8.0 or 16.8.1)?

If so, can you please put together some sort of repro that we can investigate?

[rezathecoder]

@rezathecoder Where in the docs - you should be able to use MERGE with SQL Server.

@ErikEJ Can you please give me more info about this MERGE operation ?

[rezathecoder]

@rezathecoder coming back to this, just to make sure - your original post above mentions a build time regression from 10 minutes to 30 when upgrading from VS 16.7.x to 16.8.2. Can you please confirm this (#23435 (comment) may suggest otherwise), and specifically that you're still experiencing a _regression_ in 16.8.2 (as opposed to 16.8.0 or 16.8.1)?

If so, can you please put together some sort of repro that we can investigate?

@roji Yes the slowness existed before 16.8.2 but it got worse in 16.8.2.
I temporarily solved my problem by excluding old migrations from build process

[ErikEJ]

sure: https://docs.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql?view=sql-server-ver15

[roji]

@rezathecoder what you describe doesn't correspond to anything I'm aware of at the moment... Having 250MB of source files in your migrations directory (because of seeding) is indeed going to cause significant build slowness - but there isn't supposed to be a regression.

In order for me to investigate, can you please put together a repro project?