Roslyn: Generators - analyzer using references causing problems CS8785

[mgravell]

I have an idea for an analyzer I'd like to add, reusing functionality that is already packaged in a netstandard2.0 package.

Repro is here: https://github.com/protobuf-net/protobuf-net.Grpc/pull/85/ - the interesting projects are:

• src/generators/protobuf-net.Generator (not the Grpc one!!!)
• toys/PlayContracts

In my analyzer csproj I have the two expected refs, plus a commented out one for the tool I want to use:

<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="3.6.0-3.20207.2" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.0.0-beta2.final" PrivateAssets="all" />
<!--<PackageReference Include="protobuf-net.Reflection" Version="3.0.0-alpha.143" PrivateAssets="all" />-->

The analyzer currently just lists the additional files (I have a <AdditionalFiles Include="test.proto" />). With the above reference commented out, everything kinda works; however, if I uncomment the <PackageReference> and do nothing else, it stops working completely, citing:

Severity Code Description Project File Line Suppression State
Warning CS8785 Generator 'Protogen' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. PlayContracts (netstandard2.0) C:Codeprotobuf-net.GrpctoysPlayContractsCSC 1 Active

(the actual "do the thing" code is there but commented out)

This may simply be a packaging thing, but... it isn't obvious how to fix this; I've tried numerous things - with/without the PrivateAssets hint, with <Analyzer> dll references vs project references to the analyzer, etc. It could be as simple as a dll version conflict, but I can't find any way of getting the info out to tell me what exploded.

Any guidance @chsienki ?

[jmarolf]

unfortunately nuget does not know how to pack analyzer dependencies and we've never done to work to call nuget to dynamically resolve analyzer dependencies at runtime. Effectively, you need to place the protobuf-net.Reflection contents next to your analyzer/source generator.

[chsienki]

Sounds like @jmarolf can guess what the error is. We have work to do to actually surface the errors, for now they're just swallowed unless you have a debugger on it.

Current 'best guidance' (very loosely speaking) is to Debugger.Launch() in your initialize method. Then you can attach and see what's actually failing during a compilation.

[mgravell]

@jmarolf I'm not entirely sure what that means in practice as either an author or consumer. When you say nuget doesn't know... so: how does one most correctly ship an analyzer? Is this:

1. don't have non-inbuilt dependencies, they won't work; inline everything into one assembly
2. If it can resolve one level; write your own nuspec by hand that ships everything rather than using transitive trees
3. Some exotic csproj flag I haven't used when packaging
4. Something else I haven't thought of?

Or am I misunderstanding the problem?

[davidfowl]

I assume 1.

[sharwell]

If your analyzer assembly has _unique_ dependencies (i.e. no other part of the hosting application references the dependency, and no other analyzers reference the dependency), you can place the dependency assembly in the same folder of the NuGet package where the analyzer assembly is.

If the dependency is not unique, this may or may not work, and failures may be non-deterministic. The two cases where I know this has occurred in the past both ended up treating the situation like (1): remove or inline the dependency.
https://github.com/DotNetAnalyzers/StyleCopAnalyzers/pull/2351
dotnet/roslyn#41096

[davidfowl]

Once you "do plugins" you have to solve all the same problems we solve today at build time with nuget (unification etc)

[jmarolf]

yeah, we haven't tackled runtime-dependency-resolution for analyzers. As @davidfowl says, solving this would likely need to start with additions to nuget.

[mgravell]

Oof. Right, I guess I'm going to have to forget about transitive package refs for now, and do something horrible in my build, either with some "and include these .cs files", or maybe some IL-merge/fody fun. This could get interesting.

[danielcrenna]

Was about to submit a repro for this because it seems counter-intuitive that you can't simply reference some other otherwise uncontroversial netstandard2.0 assembly in your source generator library, without it failing with an obscure warning.

I guess the least horrible thing to do here would be to post-build copy the package references to the analyzer build folder?

[danielcrenna]

It doesn't look like copying the offending dependency into the same location as the analyzer .DLL itself is working for me, maybe because System.Text.Json is implicitly referenced in the ASP.NET Core FrameworkReference.

Example code:

<ItemGroup>
    <PackageReference Include="System.Text.Json" Version="5.0.0-preview.3.20214.6" PrivateAssets="All" GeneratePathProperty="true" />
</ItemGroup>
<Target Name="CopyPrivateReferences" AfterTargets="Build">
    <Exec Command="echo f | xcopy $(PkgSystem_Text_Json)\lib\$(TargetFramework)\System.Text.Json.dll $(TargetDir)\System.Text.Json.dll /Y"/>
</Target>

[danielcrenna]

And trying to ILMerge that single dependency quickly descended into madness:

<ItemGroup>
    <PackageReference Include="System.Text.Json" Version="5.0.0-preview.3.20214.6" PrivateAssets="All" GeneratePathProperty="true" />
    <PackageReference Include="System.Memory" Version="4.5.4" GeneratePathProperty="true" />
    <PackageReference Include="System.Buffers" Version="4.5.1" GeneratePathProperty="true" />
    <PackageReference Include="System.Runtime.CompilerServices.Unsafe" Version="5.0.0-preview.3.20214.6" GeneratePathProperty="true" />
    <PackageReference Include="System.Text.Encodings.Web" Version="5.0.0-preview.3.20214.6" GeneratePathProperty="true"  />
    <PackageReference Include="System.Threading.Tasks.Extensions" Version="4.5.4" GeneratePathProperty="true" />
    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="1.1.1"  GeneratePathProperty="true" />
</ItemGroup>
<Target Name="ILMerge" AfterTargets="Build">
    <Exec Command="$(ILMergeConsolePath) /out:$(AssemblyName).dll ^
          /lib:$(NuGetPackageRoot)NETStandard.Library\2.0.1\build\netstandard2.0\ref ^
          /lib:$(PkgSystem_Runtime_CompilerServices_Unsafe)\lib\$(TargetFramework) ^
          /lib:$(PkgSystem_Buffers)\lib\$(TargetFramework) ^
          /lib:$(PkgSystem_Memory)\lib\$(TargetFramework) ^
          /lib:$(PkgSystem_Text_Encodings_Web)\lib\$(TargetFramework) ^
          /lib:$(PkgSystem_Threading_Tasks_Extensions)\lib\$(TargetFramework) ^
          $(PkgSystem_Text_Json)\lib\$(TargetFramework)\System.Text.Json.dll ^
          $(PkgMicrosoft_Bcl_AsyncInterfaces)\lib\$(TargetFramework)\Microsoft.Bcl.AsyncInterfaces.dll ^
          "/>
</Target>

[tmat]

Whatever you do, please don't use ILMerge.

[tmat]

I think the only sane solution is for Roslyn to run analyzers and code generators on .NET Core only.

[mgravell]

@tmat right now I'm thinking my best option is to hack a build that imports all the code I need directly as code, but: somewhere in this there is a "the story of package dependencies in analyzers is bad"

[danielcrenna]

@tmat Would love not to; how would you suggest I reference System.Text.Json from my source generator? It's ILMerge or copy-pasting the code off of GitHub. Neither one is good.

[jmarolf]

Currently working on a design to resolve this. Agree it needs to be solved

[danielcrenna]

Currently working on a design to resolve this. Agree it needs to be solved

Ah, good concept. I didn't think of stepping back into Roslyn to create snapshot tests, and wasn't aware of GeneratorDriver. Many thanks.

[wieslawsoltes]

I am using it like this:

  <ItemGroup>
    <!-- Take a private dependency on Newtonsoft.Json (PrivateAssets=all) Consumers of this generator will not reference it.
         Set GeneratePathProperty=true so we can reference the binaries via the PKGNewtonsoft_Json property -->
    <PackageReference Include="Newtonsoft.Json" Version="12.0.1" PrivateAssets="all" GeneratePathProperty="true" />

    <!-- Package the generator in the analyzer directory of the nuget package -->
    <None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />

    <!-- Package the Newtonsoft.Json dependency alongside the generator assembly -->
    <None Include="$(PkgNewtonsoft_Json)\lib\netstandard2.0\*.dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
  </ItemGroup>

as suggested in https://github.com/dotnet/roslyn/blob/master/docs/features/source-generators.cookbook.md#use-functionality-from-nuget-packages

[wieslawsoltes]

I am using it like this:

  <ItemGroup>
    <!-- Take a private dependency on Newtonsoft.Json (PrivateAssets=all) Consumers of this generator will not reference it.
         Set GeneratePathProperty=true so we can reference the binaries via the PKGNewtonsoft_Json property -->
    <PackageReference Include="Newtonsoft.Json" Version="12.0.1" PrivateAssets="all" GeneratePathProperty="true" />

    <!-- Package the generator in the analyzer directory of the nuget package -->
    <None Include="$(OutputPath)\$(AssemblyName).dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />

    <!-- Package the Newtonsoft.Json dependency alongside the generator assembly -->
    <None Include="$(PkgNewtonsoft_Json)\lib\netstandard2.0\*.dll" Pack="true" PackagePath="analyzers/dotnet/cs" Visible="false" />
  </ItemGroup>

as suggested in https://github.com/dotnet/roslyn/blob/master/docs/features/source-generators.cookbook.md#use-functionality-from-nuget-packages

And I had include all nuget PackageReference as dll's in analyzers/dotnet/cs.