Sdk: SDK double writes runtime files when publishing self contained apps

[fadimounir]

Here's a repro:
1) dotnet new console (make sure to target "netcoreapp3.0")
2) Edit the csproj and add a PackageReference to NewtonSoft.Json version 9.0.1.
3) dotnet publish -r win-x64

The Json library pulls netstandard1.x runtime packages, and when you run dotnet publish and reference netcoreapp3.0 runtime libraries, the list of files to publish will contain runtime assemblies from both 1.x and 3.0 runtimes, and will end up double writing some of them.

Example:
.nuget\packages\runtime.any.system.io\4.1.0\lib\netstandard1.5\System.IO.dll
.nuget\packages\runtime.win-x64.microsoft.netcore.app\3.0.0-preview4-27501-7\runtimes\win-x64\lib\netcoreapp3.0\System.IO.dll

This also causes errors when crossgen is enabled (Eventually).

The expected behavior is that the SDK should probably ensure that there's a unique copy of each assembly, and take the highest version one.

[fadimounir]

cc @nguerrera

[nguerrera]

@dsplaisted

[peterhuene]

I see the expected behavior with _HandlePackageFileConflictsForPublish:

/usr/local/share/dotnet/sdk/3.0.100-preview4-010642/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ConflictResolution.targets(77,5): message NETSDK1041: Encountered conflict between 'CopyLocal:/Users/peterhuene/.nuget/packages/runtime.any.system.io/4.1.0/lib/netstandard1.5/System.IO.dll' and 'CopyLocal:/Users/peterhuene/.nuget/packages/runtime.win-x64.microsoft.netcore.app/3.0.0-preview4-27506-13/runtimes/win-x64/lib/netcoreapp3.0/System.IO.dll'.  NETSDK1033: Choosing 'CopyLocal:/Users/peterhuene/.nuget/packages/runtime.win-x64.microsoft.netcore.app/3.0.0-preview4-27506-13/runtimes/win-x64/lib/netcoreapp3.0/System.IO.dll' because AssemblyVersion '4.2.1.0' is greater than '4.1.0.0'. [/Users/peterhuene/tmp/json/json.csproj] (TaskId:90)

Removed Item(s): 
  _ResolvedCopyLocalPublishAssets=
    /Users/peterhuene/.nuget/packages/runtime.any.system.io/4.1.0/lib/netstandard1.5/System.IO.dll
      AssetType=runtime
       DestinationSubDirectory=
      DestinationSubPath=System.IO.dll
      PackageName=runtime.any.System.IO
      PackageVersion=4.1.0

Perhaps I'm misunderstanding the issue? @fadimounir can you explain to me what you mean by "double writes"?

[fadimounir]

@peterhuene during the CopyFilesToPublishDirectory, the same destination file is copied from multiple sources. See my example above, where System.IO.dll is copied from both these locations:

• C:\Users\fadim.nuget\packages\runtime.any.system.io\4.1.0\lib\netstandard1.5\System.IO.dll

• C:\Users\fadim.nuget\packages\runtime.win-x64.microsoft.netcore.app\3.0.0-preview4-27501-7\runtimes\win-x64\lib\netcoreapp3.0\System.IO.dll

And this is because the NewtonSoft json library references older runtime bits. If you compile with /bl and look at the binlog, you'll see the double writes more clearly.

[peterhuene]

@fadimounir thanks for the explanation.

It appears that the copy targets aren't filtering out conflicts; the only dependency I see on _HandlePackageFileConflictsForPublish is for generating the deps file.

Checking to see if this is a regression from 2.2.

[fadimounir]

When I also try to repro this and look at the binlog, I see that @(_ResolvedCopyLocalPublishAssets) is equal to @(_ResolvedCopyLocalPublishAssetsWithoutConflicts). This causes the _ResolvedCopyLocalPublishAssets list to remain unchanged:
`
AfterTargets="_ResolveCopyLocalAssetsForPublish;
_FilterSatelliteResourcesForPublish">

<ResolvePackageFileConflicts ReferenceCopyLocalPaths="@(_ResolvedCopyLocalPublishAssets)"
                             PlatformManifests="@(PackageConflictPlatformManifests)"
                             TargetFrameworkDirectories="$(TargetFrameworkDirectory)"
                             PreferredPackages="$(PackageConflictPreferredPackages)">
  <Output TaskParameter="ReferenceCopyLocalPathsWithoutConflicts" ItemName="_ResolvedCopyLocalPublishAssetsWithoutConflicts" />
  <Output TaskParameter="Conflicts" ItemName="_PublishConflictPackageFiles" />
</ResolvePackageFileConflicts>

<ItemGroup>
  <_ResolvedCopyLocalPublishAssets Remove="@(_ResolvedCopyLocalPublishAssets)" />
  <_ResolvedCopyLocalPublishAssets Include="@(_ResolvedCopyLocalPublishAssetsWithoutConflicts)" />
</ItemGroup>

`

Not sure if that's expected or a bug

[peterhuene]

So the filtering does happen before the copy (I forgot to notice that _HandlePackageFileConflictsForPublish has an AfterTargets set).

If you're not seeing any conflicts, then that would explain why it's copying both. What version of the SDK are you using?

[fadimounir]

I'm building dotnet/sdk using a 10-day old master branch.

[peterhuene]

I found the problem. It's related to the fix with this PR: https://github.com/dotnet/sdk/pull/3021

By not adding RuntimePackAsset items here, this moved adding the items to _ResolvedCopyLocalPublishAssets after the conflict resolution occurs, thus they aren't being respected for conflict resolution.

I'll get this fixed.