Roslyn: Slow build while compiling Roslyn

[jasonmalinowski]

Was building Roslyn locally, and I noticed the msbuild output more or less stopped, with VBCSCompiler still spinning CPU. Waited a minute or so and it was still spinning one core, so I attached and got some dump files at various points during the build. Eventually it started going again; it seems like build time might have regressed for some project?

[jasonmalinowski]

Dumps available at \\mlangfs1\public\jasonmal\Dumps\Roslyn\41587

[CyrusNajmabadi]

I've also noticed it. It appears as if build just pauses for a wihle (though at least one core is active. Then, whatever is blocking progress finishes, and it shoots back up to being multicore.

It may be that we added some sort of dependency of a lot of downstream projects on one project/task somehow. So the system may be working as designed. However, it seems like a def regression over how the build used to zip through.

[CyrusNajmabadi]

I noticed it's happening here:

10>------ Build started: Project: Microsoft.CodeAnalysis.EditorFeatures.Test.Utilities, Configuration: Debug Any CPU ------
11>------ Build started: Project: Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests, Configuration: Debug Any CPU ------
12>------ Build started: Project: Microsoft.CodeAnalysis.CSharp.EditorFeatures2.UnitTests, Configuration: Debug Any CPU ------
13>------ Build started: Project: Microsoft.CodeAnalysis.EditorFeatures.UnitTests, Configuration: Debug Any CPU ------
14>------ Build started: Project: Microsoft.CodeAnalysis.VisualBasic.EditorFeatures.UnitTests, Configuration: Debug Any CPU ------

Despite having 5 active builds in flight, only a single core of my machine is active.

[sharwell]

This was caused by #41575. Here is the diff of the changes that were brought in by this commit:

https://github.com/dotnet/arcade/compare/b0e8d3944155f94f83deea8afe025debe369e69f...56e162725058d80918f7aa7e0e689e1fde5c2106

[jaredpar]

I'm seeing the same on my machine.

[jaredpar]

Problem appears to be compiling Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests. It's taking nearly ~5 minutes in the regression case

[mavasani]

Attaching the debugger to VBCSCompiler.exe when compiling Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests shows lot of worker threads stuck on emitting sequence points:

[jaredpar]

@mavasani that's what we've seen as well. It seems a recent change around equality caused a number of new collisions to occur here. It's being looked into.

[jasonmalinowski]

@jaredpar Does this need to be moved to compiler or duped against some other bug?

[jaredpar]

This is fixed now via #41674