Roslyn: editorconfig c# language rules doesn't trigger build failure

[vsfeedback]

I have the following C# document

using Newtonsoft.Json;
using System;
using System.Text;
using log4net;

namespace EditorConfigdemo {
public class ExampleClass { 

    public static void Main (string[] args) {

        Console.WriteLine("Hello, Message form an App");
    }

  private readonly int number;

  public ExampleClass ( int num ) {

    this.number = num;
  }

  public void PrintState( )
  {
    Console.WriteLine("State is {0}", number);
  }

        public int Calculate()
        {
            Int32 res;
            if( (number > 10 && number < 100) || number % 7 == 4 ) {                                   
                res = 1+2* ( 78 -2) *(8*4) - ((67+3) *3 );
            } else
            {
                res=42;
            }

            return res;
        }


   public String Info() {
       try {
           throw new Exception("Blam!");
       } catch (Exception ex) {
           Console.WriteLine(ex.Message);                                               
           throw;
       } finally {
           Console.WriteLine("Have we failed?");                                                                          
       }
   }
}
}

when I apply the following rules to it:

root = true

[*.*]
charset = utf-8
indent_style = space
trim_trailing_whitespace = true
insert_final_newline = true

[*.{ps,xml}]
indent_size = 2

[*.cs]
indent_size = 4 : warning
end_of_line = crlf

# Use language keywords instead of BCL types
dotnet_style_predefined_type_for_locals_parameters_members = true : suggestion
dotnet_style_predefined_type_for_member_access = true: suggestion

# Add parentheses around binary operators for clarity
dotnet_style_parentheses_in_arithmetic_binary_operators = always_for_clarity : suggestion
dotnet_style_parentheses_in_relational_binary_operators = always_for_clarity : suggestion
dotnet_style_parentheses_in_other_binary_operators = always_for_clarity : suggestion

# Prefer object/collection initialization
dotnet_style_object_initializer = true: suggestion
dotnet_style_collection_initializer = true : suggestion

# Prefer tupple names for anonymous tupples
dotnet_style_explicit_tuple_names = true : suggestion

# Use auto properties
dotnet_style_prefer_auto_properties = true : suggestion

# Use conditional expressions over return statements
dotnet_style_prefer_conditional_expression_over_return = true : suggestion

# Use simple conditional assignment for null checks
dotnet_style_coalesce_expression = true : suggestion
dotnet_style_null_propagation = true : suggestion
csharp_style_throw_expression = true : suggestion

# Use var keyword when type is apparent
csharp_style_var_when_type_is_apparent = true : suggestion

# Single statement code blocks surrounded by braces
csharp_prefer_braces = true : suggestion

# Put system namespaces first
dotnet_sort_system_directives_first = true : suggestion

# Space placement
csharp_space_after_keywords_in_control_flow_statements = true : error
csharp_space_around_binary_operators = before_and_after : error
csharp_space_after_keywords_in_control_flow_statements = true : error
csharp_space_between_method_declaration_parameter_list_parentheses = false : error
csharp_space_between_method_declaration_empty_parameter_list_parentheses = false: error
csharp_space_between_method_call_parameter_list_parentheses = false : error
csharp_space_before_colon_in_inheritance_clause = true : error
csharp_space_after_colon_in_inheritance_clause = true : error

# Statement placement
csharp_preserve_single_line_statements = false : suggestion
csharp_preserve_single_line_blocks = true : suggestion

# Open/close brace placement
csharp_new_line_before_open_brace = control_blocks, methods, properties, accessors, events, indexers, types : error

non of the rules marked with "error" action is triggered during the build in Visual Studio.

Only if I change action to error" for dotnet_style_predefined_type_for_locals_parameters_members the build fails

_This issue has been moved from https://developercommunity.visualstudio.com/content/problem/433321/editorconfig-c-language-rules-doesnt-trigger-build.html
VSTS ticketId: 775762_
_These are the original issue comments:_

Visual Studio Feedback System on 2/17/2019, 04:12 AM (4 days ago):

This issue is currently being investigated. Our team will get back to you if either more information is needed, a workaround is available, or the issue is resolved.

_These are the original issue solutions:_
(no solutions)

[sharwell]

Currently IDE code style rules do not apply during builds. We are actively working to make these analyzer available during compilation scenarios, at which point errors would cause the build to fail.

[kindermannhubert]

Suppose this work would enable usage of IDE analyzers from MsBuild alone. Is that right?

I am using ruleset file with IDE analyzer settings and I was just trying to enable this scenario and it was really messy experience.

1. I added Analyzer element inside project file with reference to Microsoft.CodeAnalysis.CSharp.Features.dll from VS. But during build it was missing references of this lib.
2. I added also the references (located also in VS folder). But those references have another version than version which is referenced by Microsoft.CodeAnalysis.CSharp.Features.dll (e.g. System.Composition.AttributedModel (1.0.31) is referenced, but 1.0.27 version is shipped with VS). There is no(?) simple way to guide MsBuild to use binding redirection.
3. Finally I downloaded Microsoft.CodeAnalysis.CSharp.Features.dll through Nuget with its dependencies and added those dependencies to Analyzer tags as well (which is thing would really like not to do). Another problem appeared - MsBuild has already loaded some assemblies by itself (from some MsBuild/???/Roslyn folder - do not remember exactly) which collided with NuGet assemblies.
4. I removed those colliding references from Analyzer references and it looked better but I was still getting some weird null reference exceptions from CSharpValidateFormatStringDiagnosticAnalyzer.
5. I gave up.

[jmarolf]

@kindermannhubert Analyzers are not allowed to reference Microsoft.CodeAnalysis.CSharp.Features.dll. Analyzers are run inside the compiler itself not the IDE. You only have access to APIs the compiler provides. Visual Studio Extensions can access apis in features.

[kindermannhubert]

And is there (will be) any way to enable IDE analyzers while building with MsBuild?

[jmarolf]

@kindermannhubert see dotnet/roslyn-sdk

1. Run Visual Studio Installer
2. Hit Modify
3. Select the Individual components tab
4. Check the box for .NET Compiler Platform SDK
5. Click on the Modify button

This is will give you several templates that will allow you to get started

This template can create a nuget package that can be installed and run via 'msbuild'

[jmarolf]

You can find some examples here

[kindermannhubert]

You probably misunderstand me. I know how to create and use custom analyzers (even from msbuild).
When I refer to IDE analyzers I mean built-in analyzers shipped with VS with name "IDExxxx" (e.g. IDE0059, IDE0060, ...).

[jmarolf]

@kindermannhubert there is no difference. Those analyzers have Id that start with "IDE" by convention but there is nothing special about most of them. Can you be more specific about what you are trying to do?

[kindermannhubert]

I've described above, but here is more detailed description.

This is the differences:

• Custom analyzers can be activated by adding following tag to csproj (I'm not interested in installing it as VS extension):

<Analyzer Include="somePathToAnalyzer\CustomAnalyzer.dll" />

• IDE analyzers do not have to be included this way. They are present automatically (while building from VS).
• Both types (custom and IDE) can be adjusted (=change default severity of analyzers) by ruleset file. Ruleset file is included by following tag in csproj:

<CodeAnalysisRuleSet>somePathToRulesetFile\MyRuleset.ruleset</CodeAnalysisRuleSet>

When working in VS it works (except for subject of this issue - when IDE analyzers has Warning/Error severity set up they are not breaking a build).

Example (simple program with error, but build succeeds):

My problem probably comes from the very same reason why the build is not broken in example above.
When I build solution just with msbuild as

msbuild MySolution.sln

no error is detected and build succeeds.

In comment above I was describing steps I tried to explicitly reference IDE analyzers from my project file. And its really not that simple because of the references of assemblies containing IDE analyzers (and maybe there are more problems).

[jmarolf]

@kindermannhubert Correct, if you want analyzers to fail your commandline build they need to be given to the commandline compiler. As you pointed out, once way to do this is to reference the dll directly

<Analyzer Include="somePathToAnalyzer\CustomAnalyzer.dll" />

Another alternative is to create an analyzer nuget package and reference it in your project file

<PackageReference Include="CustomAnalyzer" Version="1.0.0" />

However if you reference IDE assemblies in an analyzer that is run by the commandline compiler it will not work. This is by design. The compiler does not load all of Visual Studios dlls in order to build. If this scenario is important to you I would file a new issue on this repo.

[kindermannhubert]

I've tried

<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Features" Version="3.0.0" />

it doesn't do anything.
Ok, I will create new issue for this.

[jmarolf]

Microsoft.CodeAnalysis.CSharp.Features is not an analyzer dll so this will not work. It is an API library. Referencing it as an analyzer is not going to work.

[kindermannhubert]

@jmarolf Don't think so. It's full of analyzers..

[jmarolf]

@kindermannhubert It contains analyzer types but they are not setup to be consumed in any fashion other than being installed into the Visual Studio IDE.

[chrisahill]

upvote! We added a .editorconfig file to our solution and added some settings to compile as errors to find legacy code that needed to be updated. When compiled the build succeeded with no errors.

[mavasani]

@jmarolf I believe @kindermannhubert is requesting that we move all the in-built IDE analyzers into a separate assembly which can be consumed via a NuGet package, just like regular third party analyzers. Currently, the analyzers in the IDE layer (i.e. MS.CA.Features.dll) have lot of internal dependencies, primary ones being the options infrastructure which is tied to the Workspace. Hence these cannot be consumed through any NuGet package or even referencing MS.CA.Features.dll directly as an Analyzer item group in your project. @sharwell has created a separate CodeStyle project and NuGet package within Roslyn.sln, which currently only has the formatting analyzer, but is consumable as an analyzer NuGet package. He is also working on porting individual IDE style analyzers into this package, so then can be consumed as a NuGet package and enabled in CI - .editorconfig support in compiler is a critical step, and now that it is in place, we hope to start tackling one analyzer at a time to move it into this package.

[klausenbusk]

@sharwell has created a separate CodeStyle project and NuGet package within Roslyn.sln, which currently only has the formatting analyzer, but is consumable as an analyzer NuGet package.

So it should work? I tried the following code, but I don't get any build errors when running dotnet build (2.2):

dotnet add package Microsoft.CodeAnalysis.CSharp.CodeStyle --version 3.5.0-beta1-19554-03 --source https://dotnet.myget.org/F/roslyn/api/v3/index.json
cat << EOF
[*.cs]
csharp_new_line_before_open_brace = none
EOF > .editorconfig

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;

namespace ELJ
{
    public static class Program
    {
        public static void Main(string[] args)
        {
            CreateWebHostBuilder(args).Build().Run();
        }

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>();
    }
}

[pedramrezaei]

Hey folks, any progress on this? This would be extremely useful

[mavasani]

There are some design issues that need to be tackled to lay the foundation before the analyzers can be moved to a NuGet package: https://github.com/dotnet/roslyn/issues/38480

[ElteHupkes]

EDIT: For people arriving in the future, before doing anything with this comment, be sure to read the one directly below it.

@klausenbusk A comment under a related issue indicated that you also need a .ruleset file, and with that I actually got this to work for code formatting. So to summarize:

• Create a .editorconfig with the rules you'd like
• Create a CodeFormatting.ruleset in your project (name is arbitrary). My contents are:

<RuleSet Name="CodeAnalysis Ruleset" Description="This ruleset sets the severity for the Microsot.CodeAnalysis.CSharp.CodeStyle package, so it breaks on build" ToolsVersion="10.0">
    <!-- Even though the IDE will correctly show errors for rule violations in .editorconfig,
         we need to explicitly set the severity in a ruleset file to get build errors for it:
         https://github.com/dotnet/roslyn/issues/30541 -->
    <!-- It doesn't really seem to matter what I put in the AnalyzerId and RuleNamespace fields,
         as long as the rule ID is correct. -->
    <Rules AnalyzerId="Microsoft.CodeAnalysis.CSharp.CodeStyle" RuleNamespace="Microsoft.CodeQuality.Analyzers">
        <!-- IDE0055 is the formatting analyzer from Microsoft.CodeAnalysis.CSharp.CodeStyle. -->
        <Rule Id="IDE0055" Action="Error" />
    </Rules>
</RuleSet>

• Add it to your .csproj file like:

<PropertyGroup>
    <CodeAnalysisRuleSet>CodeFormatting.ruleset</CodeAnalysisRuleSet>
</PropertyGroup>

With that, my build has started to fail on code formatting errors. You could of course also configure it as Action="Warning" to just throw a warning. For showing these errors / warnings in the build it seems to respect the severity level in .editorconfig, so say you had:

csharp_new_line_before_open_brace = all:hint

Then it would not affect the build.

EDIT: By that I mean, it only shows the error level defined in the .ruleset for the properties configured as errors. If you have Action="Error" with a prop=value:warning it won't show up as a warning. Not ideal but somewhat workable for things you really don't want in your build.

[sharwell]

@ElteHupkes That comment is outdated. Starting with the 16.3 release, can now set the severity of IDE0055 (or any other rule) via a .editorconfig file.

# IDE0055: Fix formatting
dotnet_diagnostic.IDE0055.severity = error

[mavasani]

Yes, see official documentation with screenshots on how to do the same using lightbulb: https://docs.microsoft.com/en-us/visualstudio/code-quality/use-roslyn-analyzers?view=vs-2019#set-rule-severity-in-an-editorconfig-file

[ElteHupkes]

Ah! Indeed that also works. I spent a good afternoon trying to piece these things together but that part was completely not obvious to me. It's also a bit counterintuitive that you configure a rule with severity in a .editorconfig file, and then later on in that same file configure a rule with a severity for an analyzer that deals with that rule.. Anyhow, this is much easier, thanks.

Any other good resources on this subject that I missed? I'm assuming there's work being done porting other IDE rules to analyzers as well, any place I can follow progress on that or even help out?

[klausenbusk]

@ElteHupkes That comment is outdated. Starting with the 16.3 release, can now set the severity of IDE0055 (or any other rule) via a .editorconfig file.

Is that supposed to work, when I build the project with dotnet build?

$ dotnet new console -n test
$ cd test
$ dotnet add package Microsoft.CodeAnalysis.CSharp.CodeStyle --version 3.5.0-beta2-19605-02 --source https://dotnet.myget.org/F/roslyn/api/v3/index.json 
$ cat >.editorconfig << EOF
[*.cs]
csharp_new_line_before_open_brace = none
# IDE0055: Fix formatting
dotnet_diagnostic.IDE0055.severity = error
EOF
$ dotnet build
Microsoft (R) Build Engine version 15.9.20.63311 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 32,13 ms for /tmp/test/test.csproj.
  test -> /tmp/test/bin/Debug/netcoreapp2.2/test.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:00.90
$ dotnet --version
2.2.108

I'm still running 2.2 due to some packaging issue.

[mavasani]

@klausenbusk - you need compiler toolset 3.3 or greater + MSBuild/VS2019 16.3 or later for .editorconfig severity settings to be respected in CI. This is mentioned in the first line of the documentation here

[klausenbusk]

@klausenbusk - you need compiler toolset 3.3 or greater + MSBuild/VS2019 16.3 or later for .editorconfig severity settings to be respected in CI. This is mentioned in the first line of the documentation here

It isn't obvious that (Visual Studio 2019 version 16.3 and later) means compiler toolset 3.3 or greater + MSBuild/VS2019 16.3. I will wait for the packaging issue to be resolved and try a newer compiler toolset.

Note: I'm running Linux and is using Visual Studio Code and I'm not aware of any relevant documentation regarding this subject.

[andregizero]

@mavasani
So just to be clear: is it possible to trigger a build error by setting a rule in .editorconfig with VS 2019 16.4.1, .NET Core 3.1 and Microsoft.CodeAnalysis.FxCopAnalyzers 2.9.8? With "compiler toolset 3.3", are you referring to an unreleased version of Roslyn Analyzers?

[panda7789]

@mavasani
So just to be clear: is it possible to trigger a build error by setting a rule in .editorconfig with VS 2019 16.4.1, .NET Core 3.1 and Microsoft.CodeAnalysis.FxCopAnalyzers 2.9.8? With "compiler toolset 3.3", are you referring to an unreleased version of Roslyn Analyzers?

I've tested in VS19 16.4.2, C# Tools 3.4.1, all nuget packages 2.9.8 and warning from .editorconfig doesnt stop build. Am I doing something wrong?

[danstur]

@mavasani I can't get this to work either with the latest Visual Studio version (16.4.2) and I don't see anything particularly in the documentation you linked. Could you clarify what the current state is?

Having static analyzers that only work when you have a file open makes them rather pointless since you get the biggest benefits from automatically checking these things during CI.

[mavasani]

Can you clarify what warning ID(s) are you talking about? Note that IDE warnings, i.e. IDExxxx diagnostics, are not yet supported on command line build. Rest of the analyzer diagnostics are supported on command line build.

[danstur]

@mavasani I misunderstood. I thought this issue talked also enabling the other checks via a NuGet or built-in support for CI. But I guess this is handled by #38480 and seems way off into the future.

Which sadly means it's not feasible for us and I imagine most people serious about static analysis and gated check-ins in the master branch to make use of editorconfig for formatting. Sad, but at least there's an issue to track progress on this.

[sharwell]

@danstur IDE0055 (fix formatting) is available via this NuGet package:
https://dotnet.myget.org/feed/roslyn/package/nuget/Microsoft.CodeAnalysis.CSharp.CodeStyle
https://dotnet.myget.org/feed/roslyn/package/nuget/Microsoft.CodeAnalysis.VisualBasic.CodeStyle

[mavasani]

Update:

1. We have ported nearly half of the IDE analyzers to a pre-release C# CodeStyle NuGet package (VB one is here).
2. Adding a NuGet package reference to the above will ensure the ported IDE/editorconfig rules are enforced on build/CI. NOTE: You need to be on VS2019 16.3 or later for the NuGet package to work.
3. Work is in progress to port the remainder of the rules within the next few weeks.

@sharwell - do you think we are good to close this issue or should we wait till all rules are ported?

[yitzhaks]

@mavasani, @sharwell, I suggest waiting until an official package is released before closing this issue.

[sharwell]

@mavasani You can close it as long as you file new individual issues for the remaining ones. Or you can file a single Story issue listing the remaining ones.

[rohanaceres]

Update:

1. We have ported nearly half of the IDE analyzers to a pre-release [C# CodeStyle NuGet package](https://dotnet.myget.org/feed/roslyn/package/nuget/Microsoft.CodeAnalysis.CSharp.CodeStyle/3.6.0-3.20177.13) (VB one is [here](https://dotnet.myget.org/feed/roslyn/package/nuget/Microsoft.CodeAnalysis.VisualBasic.CodeStyle/3.6.0-3.20177.13)).

2. Adding a NuGet package reference to the above will ensure the ported IDE/editorconfig rules are enforced on build/CI. NOTE: You need to be on VS2019 16.3 or later for the NuGet package to work.

3. Work is in progress to port the remainder of the rules within the next few weeks.

@sharwell - do you think we are good to close this issue or should we wait till all rules are ported?

Will it work with VSCode?

[mavasani]

Will it work with VSCode?

The analyzer package is identical to any other third party analyzer package. It should work wherever you can run third party analyzers.

[bddckr]

Something to be aware of when using this, especially when moving from rulesets to editorconfig: https://github.com/dotnet/roslyn/issues/43051

[JuergenGutsch]

I just played around with the code style analyzers (Microsoft.CodeAnalysis.CSharp.CodeStyle) and it seems it is only working with a configured ruleset file. Does this make sense?

I thought the ruleset file isn't really needed and the .editorconfig is the source of truth.

Did I miss something? Or have I just tried rules that not yet moved to the new analyzers by accident?

[mavasani]

I just played around with the code style analyzers (Microsoft.CodeAnalysis.CSharp.CodeStyle) and it seems it is only working with a configured ruleset file. Does this make sense?

No, the rules work fine with .editorconfig as well. However, you will need to explicitly set severity of each IDExxxx rule ID associated with a code style by using the dotnet_diagnostic.ruleId.severity = warning format entry as explained here: https://docs.microsoft.com/en-us/visualstudio/code-quality/use-roslyn-analyzers?view=vs-2019#set-rule-severity-in-an-editorconfig-file. We understand it is a tedious task to map code styles to IDExxxx diagnostic IDs, so we are planning to implement a UI that will do this automatically for you: https://github.com/dotnet/roslyn/issues/40365#issuecomment-657685800. Tagging @mikadumont @jmarolf

[RehanSaeed]

You can also try the .editorconfig starter project that I created. All rules are configured:

https://github.com/RehanSaeed/EditorConfig

[mavasani]

@RehanSaeed I don't see any dotnet_diagnostic.ruleId.severity entries in your .editorconfig, so none of your code style rules will execute on CI/command line build, they will only run in the IDE live analysis. The core issue is explained in https://github.com/dotnet/roslyn/issues/44201. We are trying to work out a solution where users do not have to manually map code styles to IDExxxx diagnostics and add dotnet_diagnostic entries for those to get build enforcement from this NuGet package. Until then, you would need to add these to your .editorconfig to get build enforcement.

[lenniebriscoe]

@mavasani / anyone. Is there a complete list of IDEXXX rules (easily formatted rather than the documentation link above) that I can use to throw into my .editorconfig file to test Microsoft.CodeAnalysis.CSharp.CodeStyle analysers during build? Can't seem to find anything other than the documentation.

[mavasani]

Is there a complete list of IDEXXX rules

You can specify the following entries to escalate all code style rules in the package to default to a warning and that should enable all of them for build.

[*.cs]
dotnet_analyzer_diagnostic.category-Style.severity = warning
dotnet_analyzer_diagnostic.category-CodeQuality.severity = warning

You can then selectively downgrade specific rules to a lower severity (none/silent/suggestion) as per your preference. For example, see https://github.com/dotnet/roslyn-analyzers/blob/master/.editorconfig#L185-L239

[JuergenGutsch]

Great. Thanks for the detailed feedback. I will have another look :-)

[lenniebriscoe]

I am a little confused to be honest. The documentation Lists the Third party analysers however right at the bottom it talks about build errors and installing the analysers as nuget packages. What are the recommended Analyser nuget packages with rules that work with the .editorconfig in VS2019 16.6.3+ ? and which of those can be used to error at build time, please?

Microsoft.CodeAnalysis.FxCopAnalyzers : Some rules fail build?
Microsoft.CodeAnalysis.CSharp.CodeStyle : Errors fail build (IDEXXX rules - Pre-release)
StyleCop.Analyzers : Have these been superseded by Microsoft.CodeAnalysis.FxCopAnalyzers or are they meant to complement?

[mavasani]

What are the recommended Analyser nuget packages that are currently available with rules that work with the .editorconfig in VS2019 16.6.3+

You are confusing two orthogonal concepts here - analyzers (that perform the core analysis and report diagnostics) and editorconfig/rules (files to configure the behavior and semantics of analysis). I suggest you read through https://docs.microsoft.com/visualstudio/code-quality/analyzers-faq?view=vs-2019#code-analysis-versus-editorconfig and https://docs.microsoft.com/en-us/visualstudio/code-quality/roslyn-analyzers-overview?view=vs-2019 to get for information on the differences, as well as read about different analyzer packages.

[lenniebriscoe]

@mavasani perhaps I didn't explain myself well. I understand that .editorconfig holds the configuration for analyzers. I understand the packages check for different things. However, what is not clear is which part of Microsoft's recommended code quality analyzers i.e. Microsoft.CodeAnalysis.FxCopAnalyzers integrate with the build. At the moment I have a team of 5 developers with almost every configuration error turned on via the .editorconfig, Errors only seem to make themselves known some of the time whilst coding, many of them do not show at build time, developers with Resharper installed seem to have a more reliable experience both with error highlighting and during build. It doesn't feel like something we can rely on at the moment.

In a previous role we used Microsoft.CodeAnalysis.FxCopAnalyzers but with ruleset configuration and we had errors working at design time and build time reliably.

[mavasani]

However, what is not clear is which part of Microsoft's recommended code quality analyzers i.e. Microsoft.CodeAnalysis.FxCopAnalyzers integrate with the build.

All analyzers shipped in Microsoft.CodeAnalysis.FxCopAnalyzers work at build time.

Errors only seem to make themselves known some of the time whilst coding, many of them do not show at build time

Would you be able to identify a standalone repro and open a separate issue with actual and expected behaviors? Seems like you are hitting a potential bug in how editorconfig based severity configuration is affecting diagnostics in IDE typing and/or build time. It is pretty likely not an issue with the analyzer package itself, as the analyzer packages do not control the final severity of reported diagnostics, whether or not they execute at different analysis phases in IDE/command line, etc.

In a previous role we used Microsoft.CodeAnalysis.FxCopAnalyzers but with ruleset configuration and we had errors working at design time and build time reliably.

This strongly suggests you are hitting some bug in editorconfig based severity configuration support added to the Roslyn compiler and/or IDE. I would suggest filing an issue.

[wizofaus]

These are the package versions I have installed:

<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="3.7.0" />
<PackageReference Include="Microsoft.CodeQuality.Analyzers" Version="3.0.0"/>
<PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="3.0.0" />

And my .editorconfig has

[*.cs]

# IDE0051: Remove unused private members
dotnet_diagnostic.IDE0051.severity = error

Yet unused private members do not trigger build failures. Have I missed a step?

(I even tried creating a brand new console project with latest versions of everything and the same behaviour is observed)

I also tried this from package manager:

PM> Install-Package Microsoft.CodeAnalysis.CSharp.CodeStyle -Version 3.8.0-1.20330.5 -Source https://dotnet.myget.org/F/roslyn/api/v3/index.json
The 'Source' parameter is not respected for the transitive package management based project(s) ATest. The enabled sources in your NuGet configuration will be used. 
Restoring packages for C:\Dev\ATest\ATest.csproj...
  GET https://api.nuget.org/v3-flatcontainer/microsoft.codeanalysis.csharp.codestyle/index.json
  NotFound https://api.nuget.org/v3-flatcontainer/microsoft.codeanalysis.csharp.codestyle/index.json 889ms
Install-Package : NU1101: Unable to find package Microsoft.CodeAnalysis.CSharp.CodeStyle. No packages 
exist with this id in source(s): Microsoft Visual Studio Offline Packages, nuget.org

This appears to succeed:

dotnet add package Microsoft.CodeAnalysis.CSharp.CodeStyle --version 3.8.0-1.20330.5 --source https://dotnet.myget.org/F/roslyn/api/v3/index.json

But then fails to build with a large number of errors:

CSC : error CS8032: An instance of analyzer Microsoft.CodeAnalysis.UseSystemHashCode.UseSystemHashCodeDiagnosticAnalyzer cannot be created from C:\dev\.nuget\packages\microsoft.codeanalysis.csharp.codestyle\3.8.0-1.20330.5\analyzers\dotnet\cs\Microsoft.CodeAnalysis.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified..

BUT...

Using version 3.7.0-1.20208.4 did actually work!

Is there a timeline for having this published on nuget.org?

[mavasani]

@wizofaus You need VS 16.6 or later (compiler version 3.6.0 or later)

[wizofaus]

Ah ok, I had considered trying to upgrade VS, good to know. When this is published on nuget.org will it require the latest VS?

[wizofaus]

BTW it turns out you can put <Rule Id="IDE0051" Action="Error" /> into your .ruleset file and have that referred to by your .csproj and it works fine too.

But just to confirm, if we wanted to use this nuget package so it would work in any environment (e.g. ci build servers), it's necessary to explicitly run the dotnet add package command, given there's nothing in the .csproj file or anywhere else I can see that specifies where to find it?

[mavasani]

CodeStyle analyzer assemblies are now directly inserted into the .NET5 SDK (will need VS2019 16.8 Preview4 or later and .NET5 RC2 SDK or later, once both of these are released). You will not need to install any special analyzer NuGet package to enable code style enforcement in build if you are on the required .NET SDK and VS. Also note that you don't need your project TargetFramework to be net5 or later to enable this feature, you just need to be on the .NET5 SDK.

Official documentation with the steps to enable build time code style enforcement is at https://docs.microsoft.com/dotnet/fundamentals/productivity/code-analysis#code-style-analysis.

I am going to close this issue as no further feature work is required, please file separate Rolsyn issue(s) for anything that does not work as expected and doc issues at the above page if you need more clarity on documentation.

[RehanSaeed]

This thread mentions dotnet_analyzer_diagnostic.category-CodeQuality.severity but it exists in Roslyn here: https://github.com/dotnet/roslyn-analyzers/blob/master/.editorconfig#L185-L239.

I've raised https://github.com/dotnet/roslyn-analyzers/issues/4429 to document the categories of rules that we can set a severity for and also to ask if we can set a default severity for all categories.

[mavasani]

@RehanSaeed Setting default severity for all analyzer diagnostics is documented here: https://docs.microsoft.com/dotnet/fundamentals/code-analysis/configuration-options#scope