Runtime: System.Net.Http redirect errors in VS 15.3.0

[Tornhoof]

As requested by @karelz a dedicated bug to the issue mistakingly reported in dotnet/corefx#22781.

Story:

• Existing .NET 4.6 application, including System.Threading.Tasks.Dataflow 4.7.0 nuget package and System.Net.Http 4.3.2 package, redirects for System.Net.Http are configured
• Nuget offers update of Dataflow to V 4.8.0
• After update, redirects for System.Net.Http warn that I should redirect to V 4.2.0
  Note: If I don't touch the nuget update at all, no build errors occur

Steps to reproduce in a small repro:

1. Create .NET 4.6 project
2. Add reference to Nuget System.Net.Http (V 4.3.2) package
3. Add binding redirect to

      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.1.1.1" newVersion="4.1.1.1" />
      </dependentAssembly>

1. Compile (no warnings)
2. According to Project references the Assembly version of System.Net.Http.dll is 4.1.1.1
3. Add reference to one of (there are probably more packages)
   
   
   
   • System.Threading.Tasks.Dataflow nuget (V4.8.0)
   
   
   • System.Buffers (V4.4.0)
   
   
4. Compile

Expected result:

No warnings

Actual result:

Consider app.config remapping of assembly "System.Net.Http, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" from Version "4.1.1.1" [c:\users\tornhoof\documents\visual studio 2017\Projects\NetHttpRepro\NetHttpRepro\bin\Debug\System.Net.Http.dll] to Version "4.2.0.0" [C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\ref\System.Net.Http.dll] to solve conflict and get rid of warning.

Workaround:

• Do what it suggests and replace the redirect with

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>      
    </assemblyBinding>

I don't know why the build suddenly uses a library from the VS program directory and not from the NuGet packages.
Note: Of the packages released on August 11th, not all are affected, e.g. System.Diagnostics.DiagnosticSource V4.4.1 does not show the behaviour.

[EDIT] Fixed bullet points formatting to improve readability by @karelz

[karelz]

System.Threading.Tasks.Dataflow nuget 4.8.0 has likely reference on System.Net.Http 4.2.0 DLL.
@weshaggard @ericstj any idea why?

[karelz]

BTW: Latest nuget System.Net.Http 4.3.2 has assembly version 4.1.1.1 in lib\net46 directory

[ericstj]

Can you clarify if this is .NET 4.6 or .NET 4.6.1? You mention net 4.6, but I see later you might be targeting net461 based on this: MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\refSystem.Net.Http.dll

I repro'ed when targeting 4.6.1. The reason this is happening is because those packages (System.Buffers, System.Threading.Tasks.DataFlow, etc) have a netstandard2.0 build. When a netstandard2.0 assembly is referenced in the desktop project we automatically add all the assemblies needed to make it work: no more package references. When we do this we add a newer System.Net.Http.dll than what you were previously using, since that is part of netstandard2.0.

In a newer Visual Studio (15.3) you'll actually see a better warning:

The explicit binding redirect on "System.Net.Http, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" conflicts with an autogenerated binding redirect. Consider removing it from the application configuration file or disabling autogenerated binding redirects. The build will replace it with: "<bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" xmlns="urn:schemas-microsoft-com:asm.v1" />". 

In any case the fix is to remove that manual bindingRedirect and use the one that's auto-generated.

@rainersigwald might be able to tell us when MSBuild made that improvment to the warning message if you'd like to update your VS/build-environment to get it.

/cc @dsplaisted

[rainersigwald]

@ericstj That message was improved in https://github.com/Microsoft/msbuild/pull/317 for Visual Studio 2015 Update 2. But it was distinct from what @Tornhoof was reporting before that, so I suspect that your repros are not the same.

[Tornhoof]

@ericstj Sorry, yes it happens with .NET 4.6.1 and 4.7.0 (didn't test any others though, but I guess 4.6.2 is the same).
I understand your solution, which is basically equivalent to my workaround.

When a netstandard2.0 assembly is referenced in the desktop project we automatically add all the assemblies needed to make it work: no more package references. When we do this we add a newer System.Net.Http.dll than what you were previously using, since that is part of netstandard2.0.

Thank you for your explanation, but I'm not certain if that behaviour is valid or even useful. Suddenly my builds are not reproducable anymore, because suddenly the libraries are taken from the VS installation directory and not my NuGet package cache directory anymore. This basically introduces another error source to my builds, because now I need to also track down libraries from installation directories which are neither part of the full framework nor any of my NuGet packages.
Example:
You ship 15.3.1 (the next update to VS 2017 15.3) which includes System.Net.Http (.NET 4.6.1 lib V4.2.1).
Now the developer has that version installed, the build environment might not. So the developer will debug problems with a different library version than the production build.
To make certain, that I don't run into that problem, I enabled Specific Version ony my package references to make certain I don't suddenly get an unknown version.

Please rethink that automatic assembly resolving or simply publish the updated NuGet packages.

@rainersigwald Thank you for tracking down the pull request, I think @ericstj has indeed reproduced my problem, my message is just the Build Log output, which appears to be a bit different.

[mscrivo]

Been struggling with this for 3 days now, ever since the update was released. We had a previously compiling solution with all projects targeting .NET 4.6.2 with VS 2017 15.2. Many of our projects referenced System.* packages that were pulled in by various packages that were built against .NET Standard.

Upon upgrading to VS 2017 15.3, everything broke, I've had to resort to removing the System.* nuget packages (that seem to have newer versions bundled with 15.3) and removing the associated binding redirects. That mostly fixed all the errors and warnings, but I still have 2 problems left:

1. As soon as someone tries upgrading or installing a package that depends on these System.* packages, they are going to be pulled in again, and everything will break again.
2. I have one assembly that uses RazorEngine which references the Rosyln 2.0 nuget package which compiles fine, but is throwing errors at runtime now complaining that it can't find various assemblies, like System.IO.FileSystem 4.0.1.0

[ericstj]

Please rethink that automatic assembly resolving or simply publish the updated NuGet packages.

We're dead-ending all those packages with the intent that they become part of the framework again in .net 4.7.1. The goal of netstandard2 is to stop the whole package-based portability model because of all of these problems. You shouldn't need to chase down those DLLs, they are part of the build environment and should be installed on any machine that you consume netstandard2 libraries on.
@dsplaisted or @rrelyea should be able to point you to the installer that contains this support so that you can install it on your build machines. /cc @terrajobst @livarcocc

@mscrivo let us know how everything broke, but please do so in a new issue unless it is the same issue reported here. I want to make sure you have a solution that doesn't have those problems you mention and we get the right fixes into the product where necessary.

[Tornhoof]

@ericstj Thank you again for explanation. As your solution actually works, I'll close this bug.

[mscrivo]

@ericstj Thank you for the explanation. My main issue through all of this is the lack of guidance. I could not find anything in the release notes detailing the tooling changes and what should be done to move from tooling v1 to v2. It's still not entirely clear to me what the correct solution is. Are we to remove the nuget packages? or are we to keep them and just update the binding redirects to use the versions now provided with VS 15.3? Some clear upgrade documentation would have gone a long way here.

[ericstj]

We can't force folks to remove packages because they will appear as dependencies for existing libraries, which is why we have a mechanism for dealing with the conflicts that arise between the packages and the msbuild-provided libs. If you happened to directly reference any of the netstandard packages you can remove them, but chances are you're bringing them in transitively.

I believe we do have guidance for library developers that they no longer need to reference packages when targeting netstandard2.0, as well as this being the behavior or the "pack" command.

There is a long standing requirement (though I agree it's often hard to find central docs on this) that when dealing with multiple versions of nuget packages / assemblies on .NET framework bindingRedirects are required. This is the unfortunate characteristic of the desktop binder (GAC always wins, exact version of a reference or bindingRedirect is required) and nuget (doesn't control deployment, people will put your libraries in the GAC). This forces everyone building nuget packages to version their assemblies and when you ship more than one version of a package, so that you can ensure folks picking up the app local copy will get the update, and when multiple versions exist bindingRedirects are required.

We're really trying to make this better. @terrajobst @dsplaisted I think we do need a doc that describes what is going on with .NET 4.6.1 and netstandard 2.0.

[mscrivo]

Ok, so now I've updated a bunch of nuget packages, like System.Collections.Immutable to the latest, which added the netstandard2 ref to my projects, I then removed any NetStandard and NETStandard.Library.NETFramework I added and added <DependsOnNetStandard>True</DependsOnNetStandard> too my projects. Finally, I added the bindingRedirects suggested and I got to state where everything compiles with no warnings. But then, I run tests and see issues like this: System.MissingMethodException : Method not found: 'Void Microsoft.Azure.KeyVault.KeyVaultClient..ctor(AuthenticationCallback, System.Net.Http.DelegatingHandler[])' so it seems the new System.Net.Http is not compatible with the old one? Sigh

[ericstj]

DependsOnNetStandard is not required, the tools should do that automatically when they see a netstandard2.0 lib in your project.

For the test scenario: make sure your tests are using the bindingRedirects too. Depending on your test framework there are different ways to set this up. That error typically means that the calling code (app) and the called code (azure) don't agree on the version of the assembly that defines DelegatingHandler and the runtime loads them side by side and treats them as different types.

[mscrivo]

Thanks! Adding the bindingRedirects to our test projects seems to have done the trick. I wonder why VS didn't prompt to add them like it did for the other projects?

[ericstj]

I wonder why VS didn't prompt to add them like it did for the other projects?

It's because they are DLLs, so they aren't treated as "applications" by MSBuild so it doesn't think they need app.config. There are a few issues on this primary: https://github.com/Microsoft/msbuild/issues/1310, https://github.com/dotnet/project-system/issues/2692, https://github.com/dotnet/sdk/issues/1405, https://github.com/dotnet/sdk/issues/267. @tmat gave a good workaround to enable bindingRedirects in test projects here: https://github.com/Microsoft/msbuild/issues/1310#issuecomment-309596435

[mscrivo]

very nice, thank you for all the quick and helpful responses! Combining all these instructions/workarounds into a single upgrade doc would probably go a long way to avoid having other people become super frustrated by this. I can't count how many times I've had to do full cleans and rebuilds of our solution to get through this.

[vinniep79]

To help frame the audience of this proposed "single upgrade doc":

• Yesterday my solution compiled and ran correctly
• Today I upgraded to VS 15.3.0
• Today it still compiles, but I have all kinds of warnings that weren't there yesterday:
  

Warning Found conflicts between different versions of the same dependent assembly that could not be resolved.

And if I try running, I get:

System.IO.FileLoadException: 'Could not load file or assembly 'System.Net.Http, Version=4.1.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)'

This is the situation a lot of people are in right now, and since we can't downgrade to 15.2, we're stuck unless we can figure out how to resolve these issues.

[ericstj]

The reason this happens automatically on upgrade is because you get the new NuGet. NuGet made a breaking change in order to map netstandard2.0 to net461. This is essentially the price we had to pay to bring NETStandard2.0 back to net461 and not just some future .net framework. Previously net461 only supported netstandard1.4. Now net461 can suddenly start to load ns1.5, ns1.6 assemblies which are already out there (some of which have higher versions than the net461 assemblies in the same packages). To react to the breaking change we had to provide updated assemblies for net461 and later that make up netstandard. This is done through build targets.

The reason for the warnings in the solution explorer is due to this issue: https://github.com/dotnet/sdk/issues/1499

You only see that for projects using packages.config, so you can either update to packagereference, or follow the workaround I listed in the issue. The warnings are innocuous and merely present in the IDE, but can be annoying. An alternative workaround is to remove those references from your project.

The reason for the runtime error is due to missing bindingRedirects. Projects using automatic binding redirects will not see this error. BindingRedirects need to be updated because the assembly versions were updated (the ones we automatically inject). You'd actually see this bug even if we did nothing since it'll result from the nuget breaking change. The fix here is to enable automatic bindingRedirects if you can, or update your config files if you cannot.

I had a discussion with @terrajobst and we'll work on a doc or blog post to help folks out here.

[mscrivo]

@ericstj I have one last question for you. If the nuget packages are dead-ended, what's the path to getting rid of them in our codebases? As of right now, many package updates want to bring all those packages back into our solution and if you do that, you get all those warnings about the referenced component cannot be found. I'd like to get rid of them since they are not needed any longer.

[Tornhoof]

Aa post mortem from my side:
I basically followed @ericstj's recommendation:

• Removed my manual binding redirects from every project and added:

<PropertyGroup>
  <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
  <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

everywhere.
GenerateBindingRedirectsOutputType was new in my projects, see https://github.com/Microsoft/msbuild/issues/1310 for that problem.

• I converted all my projects to remove the packages.config and use PackageReference instead to get around the problem with SpecificVersion.
• Manually upgrade my NuGet assembly in TFS to 4.3 beta, to be able to restore .NET Standard 2.0 dlls (can't find the NuGet issue for that anymore).

Bottom line, quite a bit of work involved just for updating to VS 15.3 ;)

[mscrivo]

@Tornhoof yeah no kidding! What process did you use to convert projects to use PackageReference instead of packages.config?

[Tornhoof]

Wrote a program, which parses packages.config and tries to replace as many old project references as possible. Had to manually fix my Web Projects, because of some weird Import statements in the csproj.

https://gist.github.com/Tornhoof/3710df4711900339d69a1cc99abf3ec2

[mscrivo]

Thanks for sharing that!

[ericstj]

The path to getting rid of the packages completely is netstandard2.0 / frameworks that support it. You can think of those packages as the netstandard1.x representation of the API. Once projects target >= netstandard2.0, >= netcoreapp2.0, >= net461, etc the packages are no longer used. The packages may still appear in the package graph but will be removed (we don't want to force people to update), you'll never see them when using packagereference. Once the packages you use target netstandard2 then the old packages will not even appear in your package graph.

[mscrivo]

@Tornhoof Were you able to use PackageReferences with the old CsProj format?

I spent a bunch of time converting our projects to the new csproj format (which is very nice btw), but found out that VS 2017 memory consumption almost trippled with the new format. Went from using about 700MB of memory for solution on first load, to over 2.5GB. Also, CPU usage was much higher, went from having the solution ready in about 30s, to it taking seemingly 5 mins to be usable. A non-starter for us especially considering most of our devs are using under-powered laptops.

[gulbanana]

that sounds like something @davkean might be interested in

[mscrivo]

@gulbanana @davkean Would be happy to provide any detail you need. I really want to use the new format as I'm tired of having to deal with the package/bindingredirect problems as described above with 15.3+ I basically followed this guide: http://www.natemcmaster.com/blog/2017/03/09/vs2015-to-vs2017-upgrade/ and got it to the point where all our projects were building with no errors/warnings, and our application behaved as expected, but VS performance was abysmal. We have 93 projects in our solution with approx 6000 .cs files. Mostly dlls, 6 web projects and a couple exe projects in there. I've already done 1 and 2 from this: https://github.com/dotnet/roslyn/wiki/Performance-considerations-for-large-solutions I have not tried the other solutions as I'm weary having to make registry changes across the team.

[Tornhoof]

@mscrivo Yes it worked so far. I don't see a serious slowdown, but our solution is only half the size of yours. I noticed the permanent NuGet recovery for each build. Everything else I attributed to Resharper EAP

[karelz]

VS performance is valid concern. I would recommend to use VSFeedback tool as it is the right channel for such feedback.

[mscrivo]

@karelz will do, thank you

[vinniep79]

I'm still broken on my end. I went the route of updating all of my projects, but the scope became too large for what we can handle right now, and it eventually led into some functional changes regarding HttpContext that we can't bite off right now. Does anyone know of a downgrade path back to Visual Studio 2017 Update 2?

UPDATE: I ended up reverting back to Visual Studio 2015.

[natelaff]

I've read so many of these threads now and still don't understand what the correct solution is.

Had .NET 4.6.1 projects working fine in VS 2017 15.2, upgraded to 15.3 and suddenly I'm getting the Could not load file or assembly 'System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. error at runtime.

Do you need the binding redirect? Do I need to deploy System.Net.Http with my application (WinForms)?

[natelaff]

I'm still just confused on why a machine with 4.6.1 Framework installed can't load this System.Net.Http 4.2.0.0? I have zero idea what to do at this point?

[mscrivo]

@madmunsterdaddy It's strange I know. What I did was remove the nuget package references and remove the references in the projects and update the binding redirects to point to 4.2.0.0 that fixes everything with no warnings whatsoever.

The problem is that when you try upgrading a nuget package that has dependencies on those old netstandard system.* nuget packages, those broken references come back. That's why I tried upgrading to the new csproj format which fixes that problem, but unfortunately has really bad performance problems in VS, so I had to switch back and just be careful when I update nuget packages to not bring those dependencies in.

[natelaff]

Thanks @mscrivo, my issue seems to be more at runtime when PCs that have .NET 4.6.1 can't load the 4.2.0.0 version of that assembly?

[mscrivo]

Are they not getting copied to your bin folder? I see them in mine. We are on 4.6.1 as well and had no issues at runtime.

[natelaff]

Hmmm, they are, but do I need to deploy them now all of the sudden with my installer? This project doesn't reference any netstandard projects. I used to just have a assembly reference to System.Net.Http from GAC, now it keeps trying to redirect elsewhere. Seems like this just got wonky as of the 15.3 update.

[mscrivo]

@madmunsterdaddy yeah my guess is that you have to deploy them, or you force it to use the version in .NET Framework (4.0.0.0 I believe?) via binding redirect, not sure how well that will work though.

It did get wonky with 15.3 as they released the .NET core 2.0 tooling in that version which unfortunately had a big effect on .NET Framework projects that they didn't do a good job of explaining.

[natelaff]

Thanks @mscrivo! Seems kind of goofy that this happened. But I guess I'll just deploy because it makes the most sense.

[mscrivo]

@madmunsterdaddy np, another thing you could try is installing the .net core 2.0 runtime with your installer, that might provide all the right dlls and would prevent you from manually having to figure out which ones to include/exclude. Have not tried it though and I have no idea what the correct solution is here. For us, we just deploy everything that ends up in the bin folders, but ours is a web app.

[natelaff]

Hmm... that would be interesting. But Core moves at a different pace than standard and might lead to problems later? Some official guidance on this would be nice. I think I'll just have to deploy System.* from my bin now. It's not huge, just a few MB. Thanks for your help!

[alexangas]

I really appreciate NuGet and the hard work by the team. Unfortunately I've had numerous issues with it over the years and this one I couldn't get past, specifically the issue described by @madmunsterdaddy. I've spent too much time on NuGet problems and this was the last straw for me.

I've now converted my solution consisting of 20 projects over to Paket and followed the guide to convert-from-nuget, simplify, and install. I no longer have this build problem and binding redirects have been fixed as well.

This comment isn't intended to take anything away from the NuGet team and I sincerely hope things improve, as I'd prefer not to use a third party tool.

[karelz]

@arangas I understand your frustration.
However, I want to make clear one thing: This is not NuGet problem, the problem is much bigger, a combo across multiple components in the tooling - I just got 30min intro into all the pieces and history that contribute to the (unfortunate) current state.
Packet will have the same problem once it starts to honor netstandard compatibility mappings which say that netstandard15-netstandard20 are supported by net461.

I am not trying to diminish your pain. I just want to be fair to NuGet team -- this is not their fault and it is not about NuGet being worse than Packet. Using Packet is like downgrading NuGet client to older version, it would also "work". Once you upgrade Packet to newer version which supports the compatibility mappings, you will hit the same problems again. You can choose if you want to identify more durable workaround now or when that time comes again with Packet upgrade. We will be happy to help you in either case.

[alexangas]

OK thanks for that clarification @karelz, I appreciate your time! I'll definitely keep monitoring this issue and keep an eye on both technologies.

[karelz]

@alexangas sure thing.
IMO the best thing right now (for you) would be to work (in parallel) with @ericstj to point you to the known workarounds - they depend on the scenario you're in as there are several scenarios broken in slightly different ways with different workarounds. ... just my 2 cents

[alexangas]

Thank you @karelz ... To summarise my issue for @ericstj - it is a similar message to this comment except I receive it at compile time and the solution will not build. I tried to change all of my projects to .NET 4.6.2 and used NuGet to reinstall all of the packages. Unfortunately it made no difference. I also tried using the latest version of the System.Net.Http NuGet package but that seems to be an older DLL file version.

[Sc0tTyXL]

We ran into this problem on Monday, apparently one of the packages we used added some standard2.0 stuff to our csproj. It took us the whole day to find the root cause of the issue and was only able to solve the issue after reading this issue and dotnet/corefx#22781 as there was no other info flying around.

Why would you want to move packages back into the GAC? This means that if one of our dev machines is not up to date (or production server for that matter) we will run into issues that won't be reproducible on another machine. Using packages means that the product itself will be up to date and everybody will compile against the same version of code and libraries. And I assume more .dlls/packages will make this move.

It baffles me that this decision was made and I really don't understand it. To us this is a major regression and basically makes standard2.0 unusable. I really hope that this doesn't come to Framework...

Saving a little bit of diskspace, really is nothing compared to any issue this might give. And remember diskspace is cheap, time is not.

[mscrivo]

@Sc0tTyXL They didn't move them back to the GAC, they included them with VS 2017 15.3, if you dig in, you'll see where in Program Files they are being pulled from. But yes, we had to get everyone on our team to upgrade to 15.3 at the same time, including our build machines.

[cblaze22]

@ericstj did you ever get a chance to write that doc or blog on the work around with this? Reading this whole thread there doesn't seem to be a one shot answer. If this is just a bug, and will be fixed in the future can this be confirmed. This isn't a blocker for me but would like to upgrade libraries to the latest which I currently can't without running into this.

[erikbra]

Docs and a viable upgrade path and what-do-I-do-right-now-to-get-my-project-to-compile would be greatly appreciated. There are just so many rough edges on the .net core (don't even make me try mentioning the version - I simply can't ;)) tooling right now... It's really, really difficult trying to move slowly and smoothly to .net core from net461.

[ericstj]

@terrajobst created a doc as an issue here: https://github.com/dotnet/standard/issues/481