Stylecopanalyzers: Build performance concerns.

[NikGovorov]

I've got a small project that takes 3 seconds to compile without stylecop analyzers, adding them to the project increases compilation time to 13 seconds. It's not an issue for CI or manual(command line) build, but I definitely would not expect 400% percent longer builds in Visual Studio during development time especially taking into account that validation happens in background during file editing(incremental compilation) anyway. Is there any way to disable analyzers completely for VBCSCompiler when I build from Visual Studio?
I tried the following hack, it disables analyzers for debug builds, but the build still takes 5-6 seconds instead of expected 3.

        public static void HandleCompilationStart(CompilationStartAnalysisContext context)
        {
            if (context.Options.GetType().FullName == "Microsoft.CodeAnalysis.Diagnostics.WorkspaceAnalyzerOptions" || context.Compilation.Options.OptimizationLevel == OptimizationLevel.Release)
            {
                // ...
            }
        }

Thoughts?

[sharwell]

Build performance is a big concern for us.

Can you clarify what you mean by "It's not an issue for CI or manual(command line) build"? I interpreted this as "Adding StyleCop.Analyzers did not slow down CI or command line builds".

What version of StyleCop.Analyzers do you have installed?

Did you install Visual Studio 2015 Update 1 CTP, or are you running the original Visual Studio 2015 release?

Is your project and branch open source in a location where we can try out the build?

[NikGovorov]

Can you clarify what you mean by "It's not an issue for CI or manual(command line) build"? I interpreted this as "Adding StyleCop.Analyzers did not slow down CI or command line builds".

It takes exact same time(so using StyleCop.Analyzers slows down it), by 'not an issue' I meant that I'm personally OK with the fact that it's slower than in regular dev process, because my CI builds also do fxcop and resharper cli, so StyleCop.Analyzers is not the slowest thing in my build script :) and happen less frequently than builds from Visual Studio during development.

What version of StyleCop.Analyzers do you have installed?

I used latest from nuget: https://www.nuget.org/packages/StyleCop.Analyzers/1.0.0-beta013

Did you install Visual Studio 2015 Update 1 CTP, or are you running the original Visual Studio 2015 release?

No I didn't, I'm running Visual Studio 2015 release. Did they optimise something in Roslyn?

Is your project and branch open source in a location where we can try out the build?

Unfortunately it's not, but the issue can be easily reproduced on StyleCop.Analyzers code base. With StyleCop.Analyzers added it takes ~8 seconds to build StyleCop.Analyzers.csproj(with nuget related imports/targets disabled) project, and less than ~1 second without.

[qmfrederik]

Hi,

Just want to chime in on this discussion - I observed the following when moving from the StyleCop.MSBuild package to StyleCop.Analyzers:

• (Total) build time doubled (more or less) from 2 minutes to 4 minutes
• When starting Visual Studio, there's about 33% CPU usage (it's a dual-core PC) for 1,5 minute

The increase it total build time is something I don't worry about too much - we do incremental builds most of the time and it's not too big an issue if the build on the build server takes a bit more time.

The CPU usage when starting Visual Studio is more annoying - it slows down the machine. It's also noticeable when switching branches.

That said, the couple of minutes lost here and there are easily compensated for by the fixes StyleCop.Analyzers provides :-).

It's a closed source project, unfortunately, but I'm happy to run VS or MSBuild under a profiler if it helps & you provide some guidance on what procedure you'd like me to follow.

Cheers,

[sharwell]

From everything I can tell so far, StyleCop.Analyzers is actually remarkably efficient, consuming around 12% of the overall compilation time. Unfortunately, time spent in another part of the analyzer infrastructure appears to be responsible for the large increase in build times users are observing with StyleCop.Analyzers enabled. I filed dotnet/roslyn#6053 to expand the investigation.

[sharwell]

With 1.0.0-beta014 installed (commit 1b3948b5f73fbd0c8d61bd9122b4fc7612f2746e), StyleCop.Analyzers builds in 27 seconds on my machine. With 1.0.0-beta015 installed (commit d71ab2c8cf31462052c155f03b37ec1fe3e9a51b), it builds in 8 seconds. For now I'm going to consider this issue resolved, but we will continue to work on minimizing the overall footprint of this set of analyzers.

[NikGovorov]

Thanks @sharwell, very impressive improvement.

StyleCop.Analyzers(with analyzers | without analyzers):
Time Elapsed 00:00:01.59 | Time Elapsed 00:00:00.60
Time Elapsed 00:00:01.35 | Time Elapsed 00:00:00.57
Time Elapsed 00:00:01.39 | Time Elapsed 00:00:00.57
Time Elapsed 00:00:01.53 | Time Elapsed 00:00:00.54
Time Elapsed 00:00:01.46 | Time Elapsed 00:00:00.63

StyleCop.Analyzers.CodeFixes(with analyzers | without analyzers):
Time Elapsed 00:00:00.79 | Time Elapsed 00:00:00.40
Time Elapsed 00:00:00.80 | Time Elapsed 00:00:00.39
Time Elapsed 00:00:00.84 | Time Elapsed 00:00:00.40
Time Elapsed 00:00:00.83 | Time Elapsed 00:00:00.40
Time Elapsed 00:00:00.80 | Time Elapsed 00:00:00.43

But what do you think about having 'enable/disable' switch for StyleCopAnalyzers?

[sharwell]

But what do you think about having 'enable/disable' switch for StyleCopAnalyzers?

I'm trying to avoid this as much as possible because it reduces the overall effectiveness of the product. I'm fairly sure it's possible to do this by creating a new project build configuration (e.g. DebugNoAnalyzers) and either not including the <Analyzer Include="..."> items in that configuration or using a different code analysis rule set which disables all the diagnostics.

[NikGovorov]

I understand, it's definitely possible to implement it in a way you've described, but I think of a different scenario though: I would like to have analyzers working during editing files but not when I compile in Visual Studio. Previously I had StyleCop.Resharper running StyleCop in background during editing files and I had StyleCop.MsBuild tasks running StyleCop in cli/ci builds, so I didn't lose any time on analyzing during Visual Studio builds.

I totally understand that this feature probably should not be part of the official package/repository, but could we at least introduce a single method(base class or static) for all analyzers where I could put the following or similar logic:

C# if (context.Options.GetType().FullName == "Microsoft.CodeAnalysis.Diagnostics.WorkspaceAnalyzerOptions" || context.Compilation.Options.OptimizationLevel == OptimizationLevel.Release) { // ... }

Currently I have to change all analyzers' HandleCompilationStart method and it obviously will give me more merge conflicts later.

Thoughts?

[qmfrederik]

Cool, build times for my project dropped from 66s to 43s and 3.2 minutes to 2.6 minutes. Great improvement, thanks!

[sharwell]

@NikGovorov Just add the following at the end of your project file:

<Target Name="DisableAnalyzersForVisualStudioBuild"
        BeforeTargets="CoreCompile"
        Condition="'$(BuildingInsideVisualStudio)' == 'True' And '$(BuildingProject)' == 'True'">
  <!--
    Disable analyzers when building a project inside Visual Studio. Note that analyzer behavior for IntelliSense
    purposes is not altered by this.
  -->
  <ItemGroup>
    <Analyzer Remove="@(Analyzer)"/>
  </ItemGroup>
</Target>

[NikGovorov]

@sharwell, that's awesome! Does exactly what I need. Thanks heaps!

[whut]

I tried the trick with $(BuildingInsideVisualStudio), but without the other tricks of BeforeTargets="CoreCompile" and $(BuildingProject) it didn't work. Now it works great, thanks!

This trick should be documented somewhere, I'm sure that are many people with the same need of disabling StyleCop during build from VisualStudio as @NikGovorov and me.

BTW. it should work this way right inside Visual Studio, maybe to be implemented on some VS update, but at least there is this workaround now.

[sharwell]

BTW. it should work this way right inside Visual Studio, maybe to be implemented on some VS update, but at least there is this workaround now.

For users who enable "warnings as errors" and users who use a rule set file to treat specific analyzer warnings as errors, which would cause building in Visual Studio to produce different results than building from the command line. This would dramatically increase the chances that developers commit code they believe works but in reality causes the automated build for their project to fail.

This issue is sufficiently important that I am quite comfortable advising people to avoid this build customization and instead focus on continually reducing the performance overhead of the StyleCop Analyzers.

That said, I'll open a new issue to include this information somewhere in the documentation. (It's #1711.)

[whut]

blockquote {padding-left: 1ex; margin: 0px 0px 0px 0.8ex; border-left: #cccccc 1px solid;} p {margin: 0px;padding: 0px;}
I'm still sure that for users that do not set warnings as errors, such behavior would be very appreciated, be it enabled by default or not.

[sharwell]

@whut Take a look at #1711. Let me know if you think that approach will address the needs of our audience.