Sdk: Native assets are not copied to bin for net472 app if any referenced package uses netstandard2.0

[jnm2]

Full description is here: https://github.com/dotnet/symreader-converter/issues/151

I suspect this is an SDK bug in C:\Program Files\dotnet\sdk\2.1.504\Sdks\Microsoft.NET.Sdk\tools\net46\Microsoft.NET.Build.Tasks.dll; see https://github.com/dotnet/symreader-converter/issues/151#issuecomment-475344092.

My workaround for now is to add this to the csproj, but it's obviously brittle:

  <ItemGroup>
    <ReferenceCopyLocalPaths Include="C:\Users\jmusser\.nuget\packages\microsoft.diasymreader.native\1.7.0\runtimes\win-x64\native\Microsoft.DiaSymReader.Native.amd64.dll" />
  </ItemGroup>

[livarcocc]

@peterhuene can you take a look?

[peterhuene]

With the provided project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net472</TargetFramework>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.Common" Version="2.6.1" />
    <PackageReference Include="Microsoft.DiaSymReader.Converter" Version="1.1.0-beta1-62318-01" />
    <PackageReference Include="NuGet.PackageManagement" Version="4.5.0" />
    <PackageReference Include="ShellProgressBar" Version="4.0.0" />
    <PackageReference Include="System.Reflection.Metadata" Version="1.5.0" />
  </ItemGroup>

</Project>

I get Microsoft.DiaSymReader.Native.x86.dll in the build output because PlatformTarget was not specified, and the .NET Core SDK defaults to win7-x86 for the RID to use for copying locally assets.

When setting PlatformTarget to x64 with dotnet build /p:PlatformTarget=x64, I get Microsoft.DiaSymReader.Native.amd64.dll in the output instead, which is the expected behavior.

[peterhuene]

@jnm2 I'm going to close this issue as by design. With the .NET Core SDK when targeting .NET Framework TFMs, you need to specify x64 for PlatformTarget if you want the 64-bit assets instead of the 32-bit assets.

Please reopen if you disagree with my assessment. Thank you!

[jnm2]

I tried this with both <PlatformTarget>x64</PlatformTarget> and <PlatformTarget>x86</PlatformTarget> with no effect. I'll double-check.

[peterhuene]

I also used the property in the project file; shouldn't make a difference in this case.

The .NET Core SDK I was using to repro was 2.1.504 (I tried to match the version I think you're using).

[jnm2]

@peterhuene Thanks for looking at this. I should have mentioned up from that I tried <PlatformTarget>x64</PlatformTarget> and <PlatformTarget>x86</PlatformTarget> with no effect. I just double-checked and it's still not copying:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net472</TargetFramework>
    <LangVersion>latest</LangVersion>
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.Common" Version="2.10.0" />
    <PackageReference Include="Microsoft.DiaSymReader.Converter" Version="1.1.0-beta1-63314-01" />
    <PackageReference Include="NuGet.PackageManagement" Version="4.9.4" />
    <PackageReference Include="ShellProgressBar" Version="4.2.0" />
    <PackageReference Include="System.Reflection.Metadata" Version="1.6.0" />
  </ItemGroup>

</Project>

With the .NET Core SDK when targeting .NET Framework TFMs

By "with the .NET Core SDK," do you mean the SDK-style csproj? I'm building from within Visual Studio (2019 RC.1 SVC1), not the command line.

One more thing. Why is it by design to copy the native DLL when no platform target is specified unless there is a package reference with a netstandard2.0 target? It doesn't feel right to have the discrepancy.

[jnm2]

@peterhuene Ah, the number 2.1.504 came up when I used http://msbuildlog.com to try to get at the source of the problem. It appears to behave exactly the same as when building within VS2019 RC.1 SVC1.

[jnm2]

Mind if I reopen since <PlatformTarget>x64</PlatformTarget> in the project isn't making a difference for me?

[peterhuene]

Sure, I'll try to reproduce inside of Visual Studio then.

[peterhuene]

Are you able to reproduce from the command line with dotnet build, though?

[jnm2]

I hadn't tried before but yes, actually.

[jnm2]

.NET Core SDK (reflecting any global.json):
 Version:   2.2.200
 Commit:    27f814f6ba

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.200\

Host (useful for support):
  Version: 2.2.2
  Commit:  a4fd7b2c84

.NET Core SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.504 [C:\Program Files\dotnet\sdk]
  2.1.600 [C:\Program Files\dotnet\sdk]
  2.2.200 [C:\Program Files\dotnet\sdk]

[jnm2]

I see I had been running VS2017's MSBuild.exe from MSBuild Structured Log Viewer. Maybe that explains the 2.1.504.

[peterhuene]

I'm also unable to reproduce with that project (with <PlatformTarget>x64</PlatformTarget> added) in Visual Studio 2019 RC3.

What's the contents of your bin\Debug\net472 directory after you build?

[jnm2]

What's the contents of your binDebugnet472 directory after you build?

│   DevExpress nupkg rewriter.exe
│   DevExpress nupkg rewriter.exe.config
│   DevExpress nupkg rewriter.pdb
│   Microsoft.CodeAnalysis.dll
│   Microsoft.DiaSymReader.Converter.dll
│   Microsoft.DiaSymReader.dll
│   Microsoft.Web.XmlTransform.dll
│   Newtonsoft.Json.dll
│   NuGet.Commands.dll
│   NuGet.Common.dll
│   NuGet.Configuration.dll
│   NuGet.Credentials.dll
│   NuGet.DependencyResolver.Core.dll
│   NuGet.Frameworks.dll
│   NuGet.LibraryModel.dll
│   NuGet.PackageManagement.dll
│   NuGet.Packaging.Core.dll
│   NuGet.Packaging.dll
│   NuGet.ProjectModel.dll
│   NuGet.Protocol.dll
│   NuGet.Resolver.dll
│   NuGet.Versioning.dll
│   ShellProgressBar.dll
│   System.Collections.Immutable.dll
│   System.Reflection.Metadata.dll
│   System.Text.Encoding.CodePages.dll
│   System.Threading.Tasks.Extensions.dll
│   
├───cs
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───de
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───es
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───fr
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───it
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───ja
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───ko
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───pl
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───pt-BR
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───ru
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───tr
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
├───zh-Hans
│       Microsoft.CodeAnalysis.resources.dll
│       Microsoft.DiaSymReader.Converter.resources.dll
│       
└───zh-Hant
        Microsoft.CodeAnalysis.resources.dll
        Microsoft.DiaSymReader.Converter.resources.dll

Does it matter that I have this nuget.config?

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="MyGet (symreader)" value="https://dotnet.myget.org/F/symreader/api/v3/index.json" />
    <add key="MyGet (symreader-native)" value="https://dotnet.myget.org/F/symreader-native/api/v3/index.json" />
    <add key="MyGet (symreader-converter)" value="https://dotnet.myget.org/F/symreader-converter/api/v3/index.json" />
  </packageSources>
</configuration>

[peterhuene]

The nuget.config shouldn't matter.

If you don't mind, could you email me a binlog when building this project at [email protected]?

Before you do so, though, please read this document regarding sharing binlogs. Specifically, the fact that your environment variables and local file paths will be recorded in the log. If either contain sensitive information, please don't share the binlog with me.

[peterhuene]

Sorry, I didn't realize that the project provided was the "working" one. If I reference 4.2.0 of that progress bar package, I am able to reproduce.

Now that I'm able to reproduce, I'll look into it. No need for the binlog. Thanks!

[jnm2]

Oh, no! I'm sorry! Thanks!

I had the repro down to a single file:

repro.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.DiaSymReader.Converter" Version="1.1.0-beta1-63314-01" />
    <PackageReference Include="ShellProgressBar" Version="4.2.0" />
  </ItemGroup>

</Project>

[peterhuene]

So the native asset is disappearing from the restore graph for some reason. When ShellProgressBar is 4.0.0, which is netstandard1.6:

      "Microsoft.DiaSymReader.Native/1.7.0": {
        "type": "package",
        "native": {
          "runtimes/win-x64/native/Microsoft.DiaSymReader.Native.x64.dll": {}
        },
        "build": {
          "build/_._": {}
        }
      },

When we reference ShellProgressBar at 4.2.0, which is netstandard2.0:

      "Microsoft.DiaSymReader.Native/1.7.0": {
        "type": "package",
        "build": {
          "build/Microsoft.DiaSymReader.Native.props": {}
        }
      },

ResolvePackageAssets is correctly resolving an empty set of native assets since they aren't in the graph.

Oddly, build/Microsoft.DiaSymReader.Native.props has the following content:

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup Condition="Exists('packages.config') OR Exists('$(MSBuildProjectName).packages.config') OR Exists('packages.$(MSBuildProjectName).config')">
    <Content Include="$(MSBuildThisFileDirectory)\..\runtimes\win\native\Microsoft.DiaSymReader.Native.x86.dll">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>false</Visible>
      <Link>Microsoft.DiaSymReader.Native.x86.dll</Link>
    </Content>
    <Content Include="$(MSBuildThisFileDirectory)\..\runtimes\win\native\Microsoft.DiaSymReader.Native.amd64.dll">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>false</Visible>
      <Link>Microsoft.DiaSymReader.Native.amd64.dll</Link>
    </Content>
  </ItemGroup>
</Project>

Creating an empty packages.config file does get both of the files copied as content items.

I see https://github.com/dotnet/roslyn/issues/26160 that appears to be related.

What I don't understand, though, is how migrating one package reference from netstandard1.6 to netstandard2.0 would affect the native assets of another package when neither package is connected. Perhaps I'm missing some historical context here as I wasn't around in the 1.x/2.0 days.

@nguerrera @dsplaisted any ideas?

[dsplaisted]

@peterhuene I looked at some of the packages involved, and my guess is that because ShellProgressBar has a package dependency on the NETStandard.Library package for its .NET Standard 1.3 version, the "platforms" package with the RID graph (Microsoft.NETCore.Platforms) is being brought in. The .NET Standard 2.0 version of ShellProgressBar doesn't have the NETStandard.Library package dependency anymore (because we decided it shouldn't be represented that way), which also means it isn't getting the platforms package in the graph, which means it has no RID graph to understand the RIDs.

The workaround is probably to add an explicit PackageReference to the Microsoft.NETCore.Platforms platforms.

[peterhuene]

Ah, that makes sense. Thanks for clearing that up. That can be terribly confusing for users that just bump package versions and assets disappear like that.

@jnm2 adding a package reference to Microsoft.NETCore.Platforms (2.2.0 is latest) should solve the problem. This will provide a RID graph so that the NuGet can resolve the native assets correctly.

[peterhuene]

@jnm2 if this is a suitable workaround for you, please let me know and we can resolve this issue.

[jnm2]

@peterhuene This workaround is fine. If this a problem in the ShellProgressBar package, what is the best link I can give the author to explain the problem and how to fix it? (Or is this a point-in-time design problem with the SDK?)

[peterhuene]

@jnm2 it's not a problem with the ShellProgressBar package and instead the result of how netstandard libraries were being packaged for 1.x vs. 2.0.

To restore native assets, NuGet needs a RID graph. This is what tells NuGet that RID win7-x64 is a child of win-x64, which is a child of win when resolving assets, for example. The RID graph comes from the Microsoft.NETCore.Platforms package; its sole asset is the runtime.json file that defines the graph.

When you target a .NETCoreApp TFM, you get the RID graph as a transitive reference via the (implicitly referenced) Microsoft.NETCore.App package. When you target a .NETFramework TFM, you don't get a RID graph because there's no reference (transitive or direct) to Microsoft.NETCore.Platforms.

However, when you referenced a netstandard1x package, it had a transitive reference to Microsoft.NETCore.Platforms so a RID graph was found and the native assets were resolved. This transitive reference no longer exists for netstandard2.0 packages. As a result, you'll need an explicit reference on Microsoft.NETCore.Platforms if you desire to resolve native assets from your packages when targeting .NETFramework.

Does that make sense? Adding an explicit reference should solve the problem and is a necessary step when building an application with a .NETFramework TFM that desires to resolve native assets from its packages.

I'm going to close this issue by design for now. Thank you for your assistance in helping us to identify the issue.

[jnm2]

@peterhuene Thanks, that helps. The fix is not very discoverable and the package reference I'm adding isn't very self-explanatory, but I might remember next time. Is there hope for this to just work in the future?

[peterhuene]

I totally agree that the fix is not discoverable.

@livarcocc @nguerrera @dsplaisted @kathleendollard any thoughts on how we could improve upon this experience for our users? Is there documentation surrounding building .NETFramework applications with the .NET Core SDK that might need to call this out better?

[peterhuene]

Perhaps NuGet could warn that native assets were found in a package but the RID graph was not resolved?

[dsplaisted]

@peterhuene For .NET Core 3.0, we're bundling the RID graph with the SDK. We hope to be able to pass that graph to NuGet, which would mean the platforms package wouldn't need to be in the package dependency graph anymore.

[peterhuene]

And that's regardless of TFM, right? Sounds like a solution to me. Thanks!