Runtime: Exclude="Compile" hides implementation only dependencies from RAR causing missing bindingRedirects on desktop

[ericstj]

We try to avoid exposing runtime dependencies at compile time because

1. They are implementation details of the assembly that we don't want to make part of the public API contract.
2. They may be RID specific since NuGet provides no way to expose RID-specific dependencies. RID specific dependencies may be not supported on other RIDs (eg: OpenSSL, registry, WindowsIdentity, etc).

This is done here: https://github.com/dotnet/buildtools/commit/d40435b1c460416768cc53a27091e57d948be171

This is creating a problem on desktop where it's important for Compile and Runtime assets to be equal so that RAR can see them and generate the correct bindingRedirects. We do go out of our way to try to ensure this, for example we ensure that ref/lib versions match for desktop.

We should fix buildtools to avoid exclude="compile" on desktop dependency groups.

That could happen during or after PromoteDependencies.

[weshaggard]

@ericstj are you sure this will actually fix this? I tried directly referencing the latest Unsafe package in their test project and it still doesn't generate the correct binding redirects because it doesn't see conflicting versions of Unsafe, it only sees the one version. It doesn't see the assembly reference from System.Memory which has the older version so there isn't a conflict.

[ericstj]

True if the impl has a unique conflict that would not be fixed. I don’t see a good fix for that, other than somehow flowing the impls through RAR or doing more conflict forcing tricks.

[ericstj]

Perhaps we should then broaden this issue to make validation check that reference assembly and impl have the same set of assembly references for desktop assemblies? That's a bit of a hammer and doesn't really make sense in most cases but it could help fix this.

[weshaggard]

Even if we broaden the validation we need a way to make the ref match the implementation. So we would either need to publish the lib into the ref folder (perhaps stripping the IL) or have some way to artificially add the assembly reference to the ref.

[clairernovotny]

Could this be related? https://github.com/dotnet/reactive/issues/823

A user is hitting assembly load issues with ValueTuple and RxNet

[joperezr]

I don't think so, I believe that the one you pointed is the same as this one: dotnet/corefx#32610

[Eilon]

Marking as partner blocking.

[jeremymeng]

I am hitting an error in a webtest that uses aspnetcore signalr client. Since my webtest only outputs a dll there are no app.config/web.config. How could I use binding-redirects to fix the issue?

FileNotFoundException: Could not load file or assembly 'System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.

FusionLog failure

LOG: Attempting download of new URL file:///C:/Users/yumeng/source/repos/TestResults/yumeng_YU-Z230 2018-10-29 11_52_28/Out/System.Buffers.DLL. WRN: Comparing the assembly name resulted in the mismatch: Build Number 

=== Pre-bind state information ===
LOG: DisplayName = System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
 (Fully-specified)
LOG: Appbase = file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/
LOG: Initial PrivatePath = NULL
Calling assembly : Microsoft.AspNetCore.Http.Connections.Common, Version=1.0.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\QTAgent_40.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Buffers, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/TeamFoundation/Team Explorer/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/TeamFoundation/Team Explorer/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PublicAssemblies/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PublicAssemblies/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/x64/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/x64/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/x64/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/x64/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/x64/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/x64/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/TeamFoundation/Team Explorer/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/TeamFoundation/Team Explorer/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PublicAssemblies/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PublicAssemblies/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/x64/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/DataCollectors/x64/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/x64/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/PrivateAssemblies/DataCollectors/x64/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/x64/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/Extensions/V1/x64/System.Buffers/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Users/yumeng/source/repos/TestResults/yumeng_YU-Z230 2018-10-29 11_52_28/Out/System.Buffers.DLL.
WRN: Comparing the assembly name resulted in the mismatch: Build Number
LOG: Attempting download of new URL file:///C:/Users/yumeng/source/repos/TestResults/yumeng_YU-Z230 2018-10-29 11_52_28/Out/System.Buffers/System.Buffers.DLL.
LOG: Attempting download of new URL file:///C:/Users/yumeng/source/repos/TestResults/yumeng_YU-Z230 2018-10-29 11_52_28/Out/System.Buffers.EXE.
LOG: Attempting download of new URL file:///C:/Users/yumeng/source/repos/TestResults/yumeng_YU-Z230 2018-10-29 11_52_28/Out/System.Buffers/System.Buffers.EXE.

[joperezr]

@jeremymeng can you log a separate issue with a description of how your project is constructed, the error you are hitting, and a msbuild.binlog file of your build? In order to produce one, simply run the following form a developer command prompt msbuild yourProject.csproj /t:rebuild /v:m /bl. We don't know if this issue is what is impacting your project which is why we tend to instead ask for new issues to be logged. If it turns out to be the same issue, we could always dupe them to this one, but that way we avoid having a big issue tracking several different problems.

[AArnott]

The dupes that reference this issue have a lot of votes (er... thumbs up). Please prioritize this. Can we deliver before .NET Core 3.0 via the 2.x SDKs?

[joperezr]

This issue has actually been fixed now and the packages with all the fixes are available in NuGet.

Can we deliver before .NET Core 3.0 via the 2.x SDKs?

Not sure what you mean about having the fix via 2.x SDKs, the fix was applied to the affected packages directly and don't require any changes in the SDK, which means that in order to fix this, all you need to do is to upgrade to the latest stable version all the affected packages. Here is a full list of the packages that now have the fix and their versions:

System.IO.Pipelines   v.4.5.3
System.Memory  v.4.5.2
System.Net.Http.WinHttpHandler  v.4.5.2
System.Text.Encoding.CodePages  v.4.5.1
System.Threading.Tasks.Extensions  v.4.5.2

They were all shipped via NuGet 15 days ago, which is why I'm closing this issue for now. @AArnott feel free to let me know if there is any other concerns you have about this.

[AArnott]

Thanks. I guess dotnet/sdk#2547 should stay active though to track any fixes to the SDK that are appropriate, given the argument that the binding redirect generation should take the runtime assemblies into account, even when explicit ref assemblies are substituted for the compiler.