Efcore: Large compile-duration regression after upgrade from EF Core 3.1.0 to 3.1.1

[EamonNerbonne]

I have not yet tried to reproduce this on a public repository, but I hope the problem is simple enough that that's not an issue; I'll try to make a repro asap.

Reproduction is timing compilation of a 16-project solution via dotnet msbuild bla.sln -target:restore;rebuild -verbosity:minimal -p:configuration=Release.

When the only project in that solution that depends on EF core contains the line <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="3.1.0" />, compilation takes approximately 21s (with sub-second standard deviation when the initial uncached build outlier is omitted.

When that dependency line is updated to <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="3.1.1" /> the build takes approximately 30s.

The effect appears to occur even on an almost empty solution too, but we're still internally investigating whether that's statistical noise or something significant (if I find out I'll post that here).

Further technical details

EF Core version: 3.1.0 -> 3.1.1
Database provider: compile time, likely not relevant, but SQL Server
Target framework: netcoreapp3.1
Operating system: windows 10
IDE: none.

[roji]

This would probably be a result of our analyzer, and could be mitigated by #18637. It would be very helpful to have a repro project (where we could also confirm whether #18637 fixes things or not).

[ajcvickers]

@roji Did the analyzer change in the patch release?

[roji]

Not at all - I think I misread the above (interpreted to mean a regression when upgrading to 3.0/3.1 in general).

@EamonNerbonne in that case I'm doubly curious to see a repro, as nothing relevant was supposed to have changed between 3.1.0 and 3.1.1.

[ErikEJ]

VS update to 16.4.3 ??

[EamonNerbonne]

VS update to 16.4.3 ??

Sure, I'll try that; I'm running from the command line for slightly easier repeatability.

Before update:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.100
 Commit:    cd82f021f4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.100\

Host (useful for support):
  Version: 3.1.0
  Commit:  65f04fb6db

After:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.101
 Commit:    b377529961

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.101\

Host (useful for support):
  Version: 3.1.1
  Commit:  a1388f194c

(and then about a million runtimes and sdks that are also available, but shouldn't matter).

But the compilation-duration regression by updating from 3.1.0 to 3.1.1 remains the same (time in seconds, best 3/4):

3.1.0:

21.270
21.312
23.092

3.1.1:

31.404
31.530
32.62

I'll make time for a repro tomorrow!

[EamonNerbonne]

So obviously this project is tiny, so the difference isn't as bad in absolute terms, but even a plain empty console project shows a very significant slowdown (0.80s -> 1.21s) , and simply adding the efcore benchmark pocos (just the pocos!) slows down the compile with 3.1.1 much more than the compile with 3.1.0 (0.86s -> 1.36s, so around +0.06s vs. +0.15s). Given the experience with our larger repo, I expect this trend holds to larger codebases in general.

Repro repo:
https://github.com/progressonderwijs/EfCoreCompileBench

[EamonNerbonne]

It's definitely the analyzer (thanks @ajcvickers!); as a workaround, we're disabling the analyzers we don't really gain much from, which speeds up our build considerably:

<!-- ref: https://stackoverflow.com/questions/56501705/how-do-you-disable-roslyn-analyzers-when-using-msbuild-through-the-command-line --> 
<Target Name="DisableAnalyzers" BeforeTargets="CoreCompile" Condition="'$(UseAllAnalyzers)' != 'true'">
  <ItemGroup>
    <SlowAnalyzer Include="@(Analyzer)" Condition="%(Filename) == 'Microsoft.EntityFrameworkCore.Analyzers'" />
    <SlowAnalyzer Include="@(Analyzer)" Condition="%(Filename) == 'Microsoft.AspNetCore.Mvc.Analyzers' OR %(Filename) == 'Microsoft.AspNetCore.Analyzers' OR %(Filename) == 'Microsoft.AspNetCore.Components.Analyzers'" />
    <!--Worth it for us 
    <SlowAnalyzer Include="@(Analyzer)" Condition="%(Filename) == 'xunit.analyzers'" />
    <SlowAnalyzer Include="@(Analyzer)" Condition="%(Filename) == 'Microsoft.CodeAnalysis.Analyzers'  OR  %(Filename) == 'Microsoft.CodeAnalysis.CSharp.Analyzers'" />
    -->
    <Analyzer Remove="@(SlowAnalyzer)"/> 
  </ItemGroup>
  <Message Text="Removed analyzers from $(AssemblyName): @(SlowAnalyzer)" Importance="high" Condition="'@(SlowAnalyzer)'!=''"/>
  <Message Text="Retained analyzers in $(AssemblyName): @(Analyzer)" Importance="high" Condition="'@(SlowAnalyzer)'!=''"/>
</Target> 

[roji]

FYI am investigating this - will likely post back tomorrow. Am seeing some confusing/differing behavior here - could you please double-check that using different dotnet SDKs versions has no effect for you? In other words, try dropping the following global.json, and building both of your projects with both SDKs (3.1.100 and 3.1.101):

{
    "sdk": {
        "version": "3.1.101",
        "rollForward": "disable"
    }
}

[EamonNerbonne]

@roji with that global.json I'm still seeing a slowdown:

Release310Rebuild: 0.8812637200000001 +/- 0.009853977278622115 seconds (including best time 0.8650348; excluding outliers 2.7085431; 0.9197056)

Release311Rebuild: 1.3632037399999999 +/- 0.01870920890552027 seconds (including best time 1.323881; excluding outliers 1.6083101; 1.4079421)

Debug310Rebuild: 0.8540096500000001 +/- 0.012703812647646393 seconds (including best time 0.8311511; excluding outliers 0.8910411; 0.8755666)

Debug311Rebuild: 1.35856887 +/- 0.010093029353375584 seconds (including best time 1.335977; excluding outliers 1.3953917; 1.3850316)

Debug310NoOpBuild: 0.67644803 +/- 0.011966509270129667 seconds (including best time 0.6528178; excluding outliers 0.6945741; 0.6936927)

Debug311NoOpBuild: 0.67575707 +/- 0.010171011020154282 seconds (including best time 0.6533266; excluding outliers 0.6939963; 0.6898595)

Debug310BuildWithTouchedFile: 0.7485362499999999 +/- 0.00813202082845954 seconds (including best time 0.7375213; excluding outliers 0.7807155; 0.7716095)

Debug311BuildWithTouchedFile: 1.2731513799999998 +/- 0.016004660580081067 seconds (including best time 1.2415146; excluding outliers 1.2997377; 1.2968959)

Since you mention rollForward I'll guessing you're concerned about interactions with other sdks; so full dotnet --info output:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.101
 Commit:    b377529961

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.101\

Host (useful for support):
  Version: 3.1.1
  Commit:  a1388f194c

.NET Core SDKs installed:
  2.1.802 [C:\Program Files\dotnet\sdk]
  2.2.402 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]
  3.0.100-preview9-014004 [C:\Program Files\dotnet\sdk]
  3.0.100-rc1-014190 [C:\Program Files\dotnet\sdk]
  3.1.100-preview2-014569 [C:\Program Files\dotnet\sdk]
  3.1.100 [C:\Program Files\dotnet\sdk]
  3.1.101 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview9.19424.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-rc1.19457.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview9-19423-09 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-rc1-19456-20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview9-19423-09 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.0.0-rc1-19456-20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

uninstalling all the old previews doesn't make any significant difference.

[ajcvickers]

@roji @EamonNerbonne I was able to reproduce this on my machine. I am posting the detailed build logs (from dotnet build -verbosity:diag). There are some differences in the profiling data at the end, but not anything that I have the expertise to decipher.

build310.txt
build311.txt

@Pilchie This is a build perf difference between 3.1.0 and 3.1.1. Who's a good person to ask about what could be going on here?

[Pilchie]

@ajcvickers I would grab some binlogs with -bl instead of -verbosity:diag, and then maybe @rainersigwald or @nguerrera can help us figure out what is happening

[ajcvickers]

Attaching binlogs.

binlogs.zip

[rainersigwald]

@ajcvickers looking at your logs, I'm not sure I'm seeing the differences reported here. Specifically, your 310 binlog took 1.905 seconds, while your 311 binlog claims 1.090 seconds--almost twice as fast.

One thing that might help with variability here is running without restore--in the binlogs that took a significant amount of the total time and it's not necessarily helpful in the mix. But if the problem is "no-op restore is longer now" that's also useful info!

[ajcvickers]

@rainersigwald Looks like this only reproduces for clean builds. (dotnet clean, then dotnet build):

3.1.0:

C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project310>dotnet build
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 35.12 ms for C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project310\Project310.csproj.
  Project310 -> C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project310\bin\Debug\netcoreapp3.1\Project310.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.27

3.1.1:

C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project311>dotnet build
Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 35.31 ms for C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project311\Project311.csproj.
  Project311 -> C:\Repros\EfCoreCompileBench-master\EfCoreCompileBench-master\Project311\bin\Debug\netcoreapp3.1\Project311.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:04.28

Binlogs for the clean builds: binlogs_clean.zip

I also ran clean builds with -restore:False. Here are the binlogs:
binlogs_clean_norestore.zip

[rainersigwald]

I set up a local repro, and I see the delta. Poking at it.

$ hyperfine --warmup 1 "dotnet build --no-incremental --no-restore"  
Benchmark #1: dotnet build --no-incremental --no-restore
  Time (mean ± σ):      4.328 s ±  0.096 s    [User: 3.9 ms, System: 24.2 ms]
  Range (min … max):    4.180 s …  4.516 s    10 runs


$ cd ..\Project310\

$ hyperfine --warmup 1 "dotnet build --no-incremental --no-restore"  
Benchmark #1: dotnet build --no-incremental --no-restore
  Time (mean ± σ):      3.045 s ±  0.090 s    [User: 5.5 ms, System: 11.8 ms]
  Range (min … max):    2.915 s …  3.162 s    10 runs

[rainersigwald]

_Very_ likely analyzers. On my machine the 310 project spends ~200ms in Csc while the 311 project spends 1329ms. The other task times look pretty comparable (as I'd expect). I think you'll have to do the same sort of things as in #18618 to narrow it down.

[ajcvickers]

@rainersigwald We haven't made any changes to the analyzer in 3.1.1.

[rainersigwald]

@ajcvickers Ok, so the inputs to the analyzer may have changed in a detrimental way, or the analyzer might have lost optimization information?

[ajcvickers]

@roji Time for you to take over, I think. ;-)

[roji]

I did some more digging into this and discovered something very interesting. If one shuts down the dotnet SDK build server between runs, there no longer seems to be any difference between the build times. Here's a sample run on my Linux machine:

➜  Project310 git:(master) ✗ dotnet build-server shutdown; dotnet build > /dev/null; grep version ../global.json; hyperfine --warmup 3 "dotnet build --no-incremental --no-restore"
Shutting down MSBuild server...
Shutting down VB/C# compiler server...
VB/C# compiler server shut down successfully.
MSBuild server shut down successfully.
        "version": "3.1.100",
Benchmark #1: dotnet build --no-incremental --no-restore
  Time (mean ± σ):      1.169 s ±  0.083 s    [User: 1.507 s, System: 0.175 s]
  Range (min … max):    1.077 s …  1.328 s    10 runs

➜  Project310 git:(master) ✗ cd ../Project311 
➜  Project311 git:(master) ✗ dotnet build-server shutdown; dotnet build > /dev/null; grep version ../global.json; hyperfine --warmup 3 "dotnet build --no-incremental --no-restore"
Shutting down MSBuild server...
Shutting down VB/C# compiler server...
MSBuild server shut down successfully.
VB/C# compiler server shut down successfully.
        "version": "3.1.100",
Benchmark #1: dotnet build --no-incremental --no-restore
  Time (mean ± σ):      1.162 s ±  0.086 s    [User: 1.507 s, System: 0.185 s]
  Range (min … max):    1.110 s …  1.394 s    10 runs

In other words, it would seem like building with one version of the analyzer somehow leaves some state behind (at the build server), which adversely affects the perf of another run using a different version of the analyzer. @EamonNerbonne / @ajcvickers / @rainersigwald as these are weird results it would be great for someone to reproduce them.

Should I open a new issue with the Roslyn team (anyone in particular)?

PS I've seen this behavior on both Linux and Windows
PPS I also did an IL comparison between the two version (3.1.0/3.1.1) of Microsoft.EntityFrameworkCore.Analyzers.dll from the nuget packages, and apart from versioning differences they are identical.

[Pilchie]

Tagging @agocke for thoughts on the compiler server here.

[agocke]

This is expected behavior from the compiler server when the analyzer dependencies are incompatible. If you set the environment variable RoslynCommandLineLogFile to a path on your machine, the compiler server should write out a file which lists what the analyzer incompatibilities are that are preventing re-use.

[roji]

@agocke thanks for your response. I set RoslynCommandLineLogFile and got the cmdline info, but nothing that would point at a problem. There are some different interactions at the bottom of the output but nothing I can make sense of (see bottom of diff below).

diff.txt
log.fast.trimmed.txt
log.slow.trimmed.txt

First fast run:

Project311 $ export RoslynCommandLineLogFile=/tmp/log; dotnet build-server shutdown; dotnet build --no-incremental > /dev/null; rm -f /tmp/log; dotnet build --no-incremental --no-restore; mv /tmp/log log.fast.txt
(duration 77s)

Second slow run:

Project310 $ dotnet build --no-incremental > /dev/null; rm -f /tmp/log; dotnet build --no-incremental --no-restore; mv /tmp/log log.slow.txt
(duration 1m61s)

I then trimmed some header material like process/thread ID (cat log.fast|cut -c 37- > log.fast.trimmed.txt) and diffed the result.

[roji]

Note: at this point it seems pretty clear that this has nothing to do with EF Core, and that any issue here can be worked around by restarting the compiler server (dotnet build-server shutdown) or simply restart the machine after changing the dependencies from 3.1.0 to 3.1.1.

Any further investigation would be to get to the root of this rather than to fix any currently visible problem... So I'd be OK with closing unless it's valuable to keep digging.

[agocke]

Something seems a little off about those log files -- normally each line is prefixed with a PID that's used to distinguish the different clients and server from each other. It also looks like half a conversation -- I think I only see messages from the client in there.

Are you sure you shut down the server before setting the env variable? If you didn't, the server process won't inherit the environment variable and won't log anything.

Reading between the lines -- is the scenario that you're trying to reload the same analyzer assembly with a different version? That definitely won't work. You'll need to restart the server if you want to load a different version of an assembly.

[roji]

@agocke I stripped the prefixed PIDs in order to get a meaningful diff.

But yes - what I'm seeing is that when I attempt two compilations, with two different versions of the same analyzer assembly, and with the same instance of the compiler server, the second compilation is much slower. So I understand this is expected - although it's a bit unexpected to get hard-to-diagnose slowness (rather than an error, or possibly even an automated restart of the server)... Is this worth pursuing at all?

[agocke]

The short answer here is, these are all trade-offs. Diagnosing by giving an error message would be unacceptable -- the compiler server is a performance optimization and we never guarantee that it will succeed.

Automated restart could simply make things slower, since now running the build with the old analyzer is the one that's slower. There's no way to know which one is the "important" compilation.

Realistically, the server will eventually shutdown and things will reset to a new base line. This seems like it would only be an issue immediately after upgrading an analyzer package.

[roji]

Makes sense, thanks for the input @agocke!

Am going to close this. @EamonNerbonne I haven't received confirmation from you that the issue goes away after a compiler server restart or shutdown, that would be good to have (I can reopen if not).

[agocke]

This would also be a problem if you're using two different versions in the same build, but that could produce many problems in the same vein -- you would basically be defeating the cache by having an oversized working set.

[EamonNerbonne]

Sorry for the delayed response; I've been busy elsewhere (and running repeatable build-benchmarks is a bit timeconsuming).

So, I can confirm that dotnet build-server shutdown solves the issue with the repro project (https://github.com/progressonderwijs/EfCoreCompileBench): if I reverse the order in which the benchmark runs, it's the version that's compiled first that's consistently faster than the version that's compiled second, regardless of whether version 3.1.1 or 3.1.0.

Unfortunately, this does not seem to be the (primary) cause of slowdown in our larger project; we've updating everything to 3.1.1 quite a while ago, and even complete machine reboots leave the build with 3.1.1 much slower than with the analyzer disabled.

Armed with the helpful tips here, I've looked for packages within one build that have differing version, via dotnet list progress.sln package --include-transitive > deps.txt, and after some file mangling, this appears to be the complete list of packages that were resolved to differing versions:

Microsoft.CSharp: 4.7.0; 4.5.0
--
Microsoft.IdentityModel.JsonWebTokens: 5.6.0; 5.5.0
Microsoft.IdentityModel.Logging: 5.6.0; 5.5.0
Microsoft.IdentityModel.Protocols: 5.6.0; 5.5.0
Microsoft.IdentityModel.Protocols.OpenIdConnect: 5.6.0; 5.5.0
Microsoft.IdentityModel.Tokens: 5.6.0; 5.5.0
Microsoft.Win32.Registry: 4.5.0; 4.7.0
System.Dynamic.Runtime: 4.0.11; 4.3.0
System.IdentityModel.Tokens.Jwt: 5.6.0; 5.5.0
System.Reflection.Emit.Lightweight: 4.3.0; 4.6.0
System.Reflection.TypeExtensions: 4.3.0; 4.4.0
System.Text.Encoding.CodePages: 4.5.0; 4.5.1
System.Threading.Tasks.Extensions: 4.3.0; 4.5.2
System.Threading.Thread: 4.0.0; 4.3.0
System.Threading.ThreadPool: 4.0.10; 4.3.0
System.Xml.XPath: 4.0.1; 4.3.0
Microsoft.NET.Test.Sdk: 16.5.0; 15.0.0
Microsoft.TestPlatform.ObjectModel: 16.5.0; 15.0.0
Microsoft.TestPlatform.TestHost: 16.5.0; 15.0.0
System.ServiceProcess.ServiceController: 4.7.0; 4.1.0
System.Diagnostics.StackTrace: 4.0.1; 4.3.0
System.Reflection.Metadata: 1.3.0; 1.6.0

...but those aren't analyzers, so this does not (to me) explain why disabling/enabling the analyzer during build causes such a large regression. (Analyzers are included in the output of dotnet list, so this isn't a false negative).

It appears my attempt to create a small repro project was misleading and points to a different (but usually ephemeral) issue with version upgrades. I'm pretty sure we've encountered that issue too, when updating from .net 2.2 to 3.0, but it in any case appears distinct from what's causing the current issues.

[EamonNerbonne]

Is there any info I could provide that might be more actionably helpful?

[Pilchie]

Tagging @sharwell and @mavasani, who might have other suggestions for how to look into analyzer performance.

[sanjaydebnath]

As I couldn't figure out how to exclude only migrations folder to make builds faster, what I did is... follow this & move only migrations to a different project. Then add the below in the migrations project. With this the local build time reduced significantly it seems. All other projects (including the ef context project) has all analyzers enabled (fx cop, stylecop, mvc, ef core etc.) but only migration project is excluded (as its mostly generated & I can live with that).

<RunAnalyzersDuringBuild>false</RunAnalyzersDuringBuild>

[mavasani]

@sanjaydebnath You can also try using the following new feature to explicitly mark certain files or folders as generated code in .editorconfig and exclude all analyzer execution for it: https://docs.microsoft.com/visualstudio/code-quality/use-roslyn-analyzers?view=vs-2019#configure-generated-code