Runtime: Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.

[clairernovotny]

Using the latest System.Net.Http 4.1.1 as per https://github.com/dotnet/corefx/issues/9846#issuecomment-242131706, results in an exception when starting a web app that's .NET 4.6.1:

Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.

I've emailed a repro to @davidsh

---

Execution plan & status

[UPDATED by karelz]

High-level plan:
A. Revert HttpClientHandler implementation in net46 build of CoreFX back to using original .NET Framework HTTP stack instead of WinHTTP (WinHttpHandler) based stack.
B. Revise implementation of the 8 new APIs on HttpClientHandler we introduced in 4.1.0.0 OOB package so that it works accordingly for the net46 build.

Execution plan:

1. Validate feasibility of [A]
   
   
   
   • [x] a. Port HttpClientHandler from NetFX (remove WinHttpHandler build dependency for net46 build).
   
   
   • [x] b. Add APTCA (on assembly only, no security attributes should be necessary for types or methods - same as in Desktop source code).
     
     
     
     
     
     • [x] Run the SecAnnotate tool to verify the claim above - Result: Passed
     
     
     
   
   
   • [x] c. Manually test the 2 scenarios (simplified and @onovotny’s) - Result: Verified
   
   

1. Validate feasibility of [B]
   
   
   
   • [x] a. Investigate the 2 remaining APIs (DefaultProxyCredentials, MaxConnectionsPerServer) which we do not know if we can implement. - Result: They are in the bucket 4.a below.
   
   
2. Full testing of [A] implementation (cost: 1d)
   
   
   
   • [x] a. Make changes in master
   
   
   • [x] b. Test all ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume master packages on myget)
     
     
     
     
     
     • [x] Self hosting ASP.NET Core from Windows Service - validated by @annemartijn0 (here)
     
     
     
     • [x] Azure Storage API - validated by @karelkrivanek (here)
     
     
     
     • [x] Raven.Database package + usage of new HttpClientHandler - validated by @jahmai (here)
     
     
     
     • [x] Direct dependency on System.Net.Http - validated by @pollax (here)
     
     
     
     • [x] 4.6 console app depending on System.Net.Http - validated by @MikeGoldsmith (here)
     
     
     
     • [x] 4.6 Azure webjob (console app) with ServiceBus - validated by @chadwackerman (here)
     
     
     
     • [x] 4.6 Azure Batch app - validated by @chadwackerman (here)
     
     
     
     • [ ] Not-yet-described scenario by @dluxfordhpf
     
     
     
   
   
3. Full implementation and testing of [B]
   
   
   
   • [x] a. Decide on design of 4 APIs (CheckCertificateRevocationList, SslProtocols, DefaultProxyCredentials, MaxConnectionsPerServer) which we can’t implement “correctly” - we have 3 choices - either throw PlatformNotSupportedException, or do nothing, or set the properties domain-wide instead of per-HttpClient-instance
     
     
     
     
     
     • [x] i. Implement the decision (cost: 2d)
     
     
     
     • [x] ii. List all libraries on NuGet (e.g. WCF?) which use the APIs we will be technically breaking, contact them
     
     
     
     • 4 NuGet libraries potentially affected - owners contacted - see details and tracking progress
     
     
     
   
   
   • [x] b. Implement 5 APIs which we know how to implement (cost: 3d)
   
   
   • [x] c. Final testing on master branch package - covered by [3.b]
   
   
4. Ship final packages
   
   
   
   • [x] a. Port changes into release/1.1.0 branch
   
   
   • [x] b. Produce new package - 4.3.1
   
   
   • [ ] c. Test most of ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume 4.3.1 stable package from myget feed - see here)
     
     
     
     
     
     • [ ] Self hosting ASP.NET Core from Windows Service - @annemartijn0
     
     
     
     • [x] Azure Storage API - @karelkrivanek
     
     
     
     • [ ] Raven.Database package + usage of new HttpClientHandler - @jahmai
     
     
     
     • [x] Direct dependency on System.Net.Http - @pollax (here)
     
     
     
     • [ ] 4.6 console app depending on System.Net.Http - @MikeGoldsmith
     
     
     
     • [ ] 4.6 Azure webjob (console app) with ServiceBus
     
     
     
     • [ ] 4.6 Azure Batch app
     
     
     
     • [ ] Not-yet-described scenario by @dluxfordhpf
     
     
     
     • [x] KeyVault - @Vhab (here)
     
     
     
     • [x] SignalR - @tofutim (here)
     
     
     
     • [x] OwlMQ - @JoeGordonMVP (here)
     
     
     
   
   
   • [x] d. Publish package on nuget.org - https://www.nuget.org/packages/System.Net.Http/4.3.1
   
   

---

Impact of the change - Breaking changes

Here's list of technical breaking changes caused by the proposed solution. It includes workarounds for each.
Note that these new behaviors are specific when running on net46 / Desktop. When you run on .NET Core, the behavior is intact.

1. HttpClientHandler.CheckCertificateRevocationList (introduced in System.Net.Http 4.1)
   
   
   
   • New behavior: Throws PlatformNotSupportedException
   
   
   • Workaround: Use ServicePointManager.CheckCertificateRevocationList instead (impacts the whole AppDomain, not just single HttpClientHandler as it did in System.Net.Http 4.1-4.3)
   
   
2. HttpClientHandler.SslProtocols (introduced in System.Net.Http 4.1)
   
   
   
   • New behavior: Throws PlatformNotSupportedException
   
   
   • Workaround: Use ServicePointManager.SecurityProtocol instead (impacts the whole AppDomain, not just single HttpClientHandler as it did in System.Net.Http 4.1-4.3)
   
   
3. HttpClientHandler.ServerCertificateCustomValidationCallback (introduced in System.Net.Http 4.1)
   
   
   
   • New behavior: Works fine, except that the first parameter of type HttpRequestMessage is always null
   
   
   • Workaround: Use ServicePointManager.ServerCertificateValidationCallback
   
   

4. HTTP/2.0 support (introduced in System.Net.Http 4.1)

   • New behavior: System.Net.Http (for net46 = Desktop) no longer supports HTTP/2.0 protocol on Windows 10.
   • Workaround: Target System.Net.Http.WinHttpHandler NuGet package instead.

   • Details:

     • HTTP/2.0 support is part of the new CoreFx HTTP stack which on Windows is based on WinHTTP. The original HTTP stack in .NET Framework 4.6 did not support HTTP/2.0 protocol. If HTTP/2.0 protocol is needed, there is a separate NuGet package, System.Net.Http.WinHttpHandler which provides a new HttpClient handler. This handler is similar in features to HttpClientHandler (the normal default handler for HttpClient) but will support HTTP/2.0 protocol. When using HttpClient on .NET Core runtime, the WinHttpHandler is actually built-in to HttpClientHandler. But on .NET Framework, you need to explicitly use WinHttpHandler.
     • Regardless of whether you are running using .NET Framework runtime (with WinHttpHandler) or .NET Core runtime using HttpClientHandler (or WinHttpHandler), there are additional requirements in order to get HTTP/2.0 protocol working on Windows:
     • The client must be running on Windows 10 Anniversary Build (build 14393 or later).

     • The HttpRequestMessage.Version must be explicitly set to 2.0 (the default is normally 1.1). Sample code:
       ```c#
       var handler = new WinHttpHandler();
       var client = new HttpClient(handler);
       var request = new HttpRequestMessage(HttpMethod.Get, "http://www.example.com");
       request.Version = new Version(2, 0);

       HttpResponseMessage response = await client.SendAsync(request);
       ```

[ericstj]

I happened to look at this today from another report. /cc @ChadNedzlek

This looks like it might be due to missing security attributes on the out-of-band System.Net.Http.dll compared to the inbox System.Net.Http.dll. Inbox version has AllowPartiallyTrustedCallers. So does the inbox System.Net.Http.WebRequest.dll. This means everything is treated as SecurityTransparent unless annotated otherwise.

The OOB System.Net.Http.dll is missing AllowPartiallyTrustedCallers, which makes everything security-critical. Now when the inbox WebRequest.dll loads the OOB System.Net.Http.dll it violates the security rule, since WebReqeuestHandler is transparent, but HttpClientHandler which it derives from is critical.

My repro:

1. File > new desktop application
2. Project properties > signing > enable strongname signing
3. Add reference to System.Net.Http.WebRequest
4. Install System.Net.Http 4.1.0.
5. In the main, just call new WebRequestHandler();

[fabiodiluca]

ericstj is right, I have the same problem here. This is a critical issue on System.Net.Http 4.1.0 that should be repaired. I can't use this library with .net4.6.1 because it lacks consistency.

[dluxfordhpf]

This problem is a significant pain to deal with, and in particularly makes using the Azure KeyVault client painful for my team. The only painless alternative is to downgrade to .NET 4.5.2, which is not acceptable.

Also, the workaround listed previously is insufficient. If you want to use NET 4.6.x, what we've found is you have to do the following this for it to work reliably: disable dependency checking, downgrade System.Net.Http, edit the CSPROJ and disable automatic binding redirect, modify the app.config, and typically clean/exit VS/and rebuild as I've often seen System.Net.Http in use, even for trivial projects. Here's the procedure that reliably fixes this.

1. Right click the project in Visual Studio and select Manage NuGet Packages.
2. Goto the Installed tab.
3. Scroll down to SYSTEM.Net.Http – not Microsoft.Net.Http.
4. In the right hand pane, if Installed is not 4.0.0.0, then set Version to 4.0.0.0.
5. In the right hand pane, set Dependency behavior to Ignore Dependencies. If you do NOT do this, the Microsoft.IdentityModel.Clients.ActiveDirectory package will be substantially downgraded, which is NOT correct.
6. Click Update - this button should really be labeled "Change To Version".
7. In the Preview window, verify that the ONLY change listed is "Installing System.Net.Http". If you forgot to set Dependency behavior correctly, the additional change will be listed here.
8. Click OK/Yes/I Agree on the confirm dialogs and wait for processing to complete. When complete, the package listing will show two version numbers – the check mark is next to the one that is in use.
9. On the NuGet Installed tab, select the listing for Microsoft.IdentityModel.Clients.ActiveDirectory.
10. In the right-hand details pane, set Dependency behavior to "Ignore". If you do not do this, any subsequent NuGet adds will fail when NuGet's validation for this package runs.
11. In Visual Studio, select File|Save All.
12. Open the CSPROJ file for this project in a text editor.
13. In /Project/PropertyGroup*1 (the first PropertyGroup element of Project), add the following line, or change the value of this element if it already exists:
    <AutoGenerateBindingRedirects>false</AutoGenerateBindingRedirects>
14. Save the file, then reload the project in Visual Studio.
15. Open the app.config file for this project.
16. Find the line for System.Net.Http and edit it to redirect it to 4.0.0.0 instead of 4.1.0.0. So find this:

<dependentAssembly> 
<assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> 
<bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.1.0.0" /> 
</dependentAssembly> 

And change it to this:

<dependentAssembly> 
<assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> 
<bindingRedirect oldVersion="0.0.0.0-4.1.0.0" newVersion="4.0.0.0" /> 
</dependentAssembly> 

1. Rebuild the project. If you get an exception when running Azure Key Vault code, one or more *.config files in the bin/debug or similar directories have not been updated. You may have to exit Visual Studio to clear them and rebuild.

[fabiodiluca]

dluxfordhpf, thanks for your time explaining how you have worked this out. Redirecting to System.Net.Http 4.0 worked for me in .net4.6.1, But is still very hard to maintain (the nuget dependency). Looking forward the version that will fix this.

[ericstj]

Redirecting could cause problems if folks are using any of the new API added in System.Net.Http 4.1.0.

in particularly makes using the Azure KeyVault client painful for my team

@ChadNedzlek had the same problem and was able to workaround it by passing in an HttpClient he created himself to the KeyVaultClient constructor. If you don't use WebRequestHandler you'll avoid the problem.

EG:

c# var handler = new HttpClientHandler(); // configure handler // eg: handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => errors == SslPolicyErrors.None; var client = new HttpClient(handler); var kvclient = new KeyVaultClient(async (authority, resource, scope) => "foo", client);

@davidsh I think you need to put AllowPartiallyTrustedCallers back on System.Net.Http.dll. /cc @morganbr

[guillaumeraymond]

@dluxfordhpf Big thanks for the steps.
It is temporary and we hope a fix soon but still, I can continue to work on the project !

[robertmclaws]

@terrajobst This is a blocking production issue. Any idea when we can get a fix onto NuGet? Thanks!

[b3nt0]

Just ran into this myself. Would be awesome if we didn't descend into dependency hell in .Net. It's heading that way.

EDIT: Fixed with a prior comments suggestion to use the system httpclient??
new KeyVaultClient(GetAccessToken, new HttpClient(new HttpClientHandler()))
That seemed to get it...

[MichaelVach]

The problem is only becoming worse as more and more nuget packages from Microsoft take a dependency on the latest version of System.Net.Http. I am probably not the only one who constantly upgrades his Microsoft nuget packages to the latest pre-release version. For example packages that no longer work for me in their latest version:

Microsoft.IdentityModel.Clients.ActiveDirectory
Microsoft.TeamFoundationServer.Client
Microsoft.VisualStudio.Services.Client.
....

[robertmclaws]

I still don't understand why this package is still available. @terrajobst @coolcsh can we get this package taken down/fixed? It's REALLY causing issues with complex app environments. Wasted several hours trying to keep the offending package from creeping into the build. Thanks!

[dluxfordhpf]

We are looking at binding to the System.Net.Http in the NET Framework instead of this broken thing from NuGet. I agree, this problem is ridiculous, and very expensive to deal with as you must synchronize NuGet versions between projects, prevent auto binding updates, and fix/check your app.config. What surprises me is that it’s in such a widely used assembly. Perhaps MS doesn’t really care about KeyVault that much?

[guillaumeraymond]

It is also used by the ActiveDirectory Nugget

[sandersaares]

I have reverted some libraries from targeting .NET Standard due to this and related issues where broken NuGet packages just mess up apps that target .NET Standard. This is a sorry state of things.

[terrajobst]

Thanks for filing. We're actively looking into this. Stay tuned.

[CodingGorilla]

This has become a significant issue for me; we use a lot of our own nuget packages internally. And it seems nuget just _will not_ let those bindingRedirects alone. Everytime we update one of our internal packages, it changes the redirect back to newVersion="4.1.0.0". Is there a way to stop it from doing that, or is there a fix on the horizon?

[mdsharpe]

Encountered when running application over HTTPS, which worked fine over HTTP. Not sure if any other significant differences.
Workaround of setting newversion="4.0.0.0" worked for me.

[jahmai]

Still an issue in NETStandard 1.6.1. Why?!

[LoulG]

System.Net.Http 4.3.0 is out. Someone tried ?

[jahmai]

@LoulG Yep, still a problem I'm afraid.

[robertmclaws]

I spoke with @terrajobst on Twitter, and he said it's actually a bigger problem, and they have like 10 people working on it now. I'm personally not sure why they are not pulling the versions of this package displaying the problem, as I thought that's what package management was for. But the next we can do at this point is wait.

[odinnou]

@LoulG same here, I have updated all my Nuget Packages, with the release of .net core 1.1 and i got this issue

System.TypeLoadException: Inheritance security rules violated by type: 'System.Net.Http.WebRequestHandler'. Derived types must either match the security accessibility of the base type or be less accessible.

Firstly, I think it was due to IdentityServer/IdentityServer3.AccessTokenValidation but, with reading this issue I understand my situation t_t, I spent several hours to try to solve it

EDIT :
OMG,
Workaround of setting newversion="4.0.0.0" worked for me too

I try to update to 4.3 but, same issue :(((

[HalidCisse]

same issue here after upgrading

[vascofernandes]

Issue still present with 4.3.0.

[Kwal]

@terrajobst can you provide any updates on this issue or any further insight into the deeper problem that @robertmclaws referenced?

[matthewrwilliams]

Any reason why this fix works locally, but when deployed to an Azure Cloud Service, the error persists?

[jahmai]

Sometimes packaging Azure can screw up your binding redirects, try unzipping the cspkg and see what is actually being deployed.

[matthewrwilliams]

Here is the version of the System.Net.Http.dll

I've been working on this for a few days now. Was super happy to find this fix and in tears when deploying out to Azure.

[jahmai]

What about the .config file containing binding redirects for the project? I actually expect it to still deploy the broken version of the assembly due to CopyLocal being true from the nuget package, but the binding redirect should ensure that the framework version of the assembly is instead loaded inside the cloud service VM.

[matthewrwilliams]

It is!!! WTH?

<dependentAssembly> <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" /> </dependentAssembly>

Looking back into my project, the web.config has reverted as well. I'll have to pick this up again in the morning. Thanks for some leads!

[jahmai]

I find that if you use AutoGenerateBindingRedirects and then modify any nuget package for the project, it will "fix" your previously hand-modified redirects back to the broken version. Very frustrating.

[robertmclaws]

But part of the problem is, if you DON'T use AutoGenerateBindingRedirects when you add new packages, you might run into other issues too.

I've been dealing with this nonsense for over a week now while having to deploy a new version of our app. Best I can suggest is to get in the habit of checking your web.config every time you deploy.

[dluxfordhpf]

Yeah, you have to both disable binding updates via hand edit in the CSPROJ, and then fix the .config binding redirects, AND reconfigure NuGet in the GUI to NOT update dependencies, and THEN you’re ok. Yes, this is a major PITA, I am surprised it has been out in production so long. My team has been cursing at it regularly for months now.

[matthewrwilliams]

Unfortunately following the above still only fixes my dev environment and not azure prod. I checked the latest pub file and the web.config was properly set along with my dll that is published being the version pictured above. Unfortunately, my issues is pertaining to the Azure Search library, so this fix was proving to be promising. Any other ideas out there? At a bit of a loss. Thanks for the help.

[jasoncouture]

Why is this over 2 months old? This is a huge issue. Please fix.

[ciel]

This is completely a stop-ship issue, it absolutely needs to get addressed.

[chadwackerman]

For f#$@'s sake guys, fix this already. This is idiocy.

[chadwackerman]

@suprduprmn I updated and consolidated all packages and then changed this in all the app.config and web.config files:

<bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.0.0.0" />

Only then would my web app launch on Azure and be able to make https calls to Azure services that depend on System.Net.Http. YMMV but good luck.

And @terrajobst ... is there a proper place I can formally complain about neglecting major issues like this? The happy-go-lucky rules of open source don't apply here. This is Microsoft Showstopper 101 stuff circa 1995. There's no way "the community" can help. You gotta fix it and you gotta architect tools and process to make sure it stops happening. I'm seeing stuff like this in multiple Microsoft projects and it's totally unacceptable. It's obvious that there are huge gaps in testing. Basic scenarios simply don't install or build clean let alone function properly at runtime.

[karelz]

I want to apologize for the XXXL long time it took us to respond to this issue. It has been 1 month since last response and the issue is opened for 3+ months. I agree that this is not acceptable for a high-impact issue like this.

It was brought up to my attention this morning and I tried to do some historical digging for last few hours. The good news is that the right folks are looking into the problem for last few weeks (maybe even longer, I didn't check all the history). Sadly, the solution is very complicated :( and that's why we do not have good answer or ETA yet (although it is not an excuse for lack of update from our side).
There are couple of potential solutions, but even the experts are not in agreement yet and there are downsides to each of them.
We will start daily sync ups on the issue next week, trying to get solution for the problem as fast as possible. I will post updates (hopefully with more technical details) when we have them, at least on weekly basis.

Sadly, this is the best I can do at this point (i.e. sincere apology and assurance we are and will treat it seriously and do our best to fix it ASAP). If anyone has alternative suggestions, I am all ears.

[jahmai]

@karelz I wouldn't want to distract the team from actually fixing the problem right now, but personally I'd love to hear a post-mortem analysis on the root cause of this issue and how it made it's way through QA. This has caused major headaches and I think transparency would earn some trust.

[karelz]

@jahmai Understood, I am also interested in that, but I want to focus on solution first. Then we can analyze what happened why and how to prevent such mishaps in future.

[robertmclaws]

Thank you for the response. I think the best solution at the moment is to deactivate any package that doesn't point back to System.Net.Http 4.0.0. There are at least 2 versions of the package that are bad. Isn't that the point of distributing this stuff through NuGet in the first place?

[PureKrome]

hugs @ ms-team

[robertmclaws]

Also, just so you know, between this issue, the issues around HttpClient being so poorly designed, the issues around Microsoft.Security.Owin.Jwt being broken against current dependencies, and the state of .NET Core...

It is an incredibly frustrating time to be a .NET developer right now. Each deployment now requires 30 minutes of checking assembly binding references so my apps don't get broken in production. This is not the Microsoft I've championed for so long. I love you guys, and I don't want to be harsh at all... But it feels like some tough love is needed to restore the quality status-quo.

Whatever has to be done. Even if it means shipping a 4.3.1 that references the old 4.0 package. Please, just do it soon.

[dluxfordhpf]

Thanks guys. FWIW, if you need to make a breaking change, please do so. We dislike this, but we’ve been living with the pain for several months, and now that we know you’re engaged, we can wait a couple more, even if we have to make some API changes.

[chadwackerman]

Something is off the rails here. I've been shipping C# apps for 15 years. Ironically as higher level abstractions are being shipped by Microsoft I'm spending more and more time digging deep into internals I never previously had to worry about. YOU'RE DOING IT WRONG.

I'm not smart enough to understand why reverting the trust flags on this library is a huge deal nor do I understand why I don't have more control over that from my app anyway.

If I wanted to be surprised at runtime by random library and package-management induced errors that the compiler didn't catch, I'd write my apps in Ruby.

[karelz]

Quick update:
We have met several times this week. We brainstormed the options and figured out funding.
@CIPop is working on this issue as top priority (work transitioned from @davidsh who will be OOF starting next week for the rest of the year).

Status:

• We were able to reproduce @onovotny's original repro and craft small repro.
• We are pursuing solution [3] below - focusing its remaining open question, i.e. validating technical feasibility of the option.
• We are scouting in parallel solution [2] below - scouting its open question, i.e. assessing the impact of the solution on ecosystem.

Problem description

• We shipped System.Net.Http 4.3.0.0 “OOB” in June
  
  
  
  • Notable content (for this context): HttpCient, HttpClientHandler
  
  
  • Customer value: Certificates support, http2 support, WinHttp stack on Desktop
  
  
  • All platforms except Desktop work fine - for Desktop the API surface does not have SecurityAttributes for PartialTrust code (APTCA = AllowPartialTrustCodeAttribute)
  
  
  • On Desktop, the OOB library overrides usage of System.Net.Http 4.0.0.0 which is part of the platform (and has APTCA)
  
  
• System.Net.WebRequestHandler 4.0.0.0 is part of Desktop
  
  
  
  • It depends on System.Net.Http and has APTCA, therefore requires System.Net.Http to have APTCA
  
  
  • It uses internal types from System.Net.Http (which has InternalsVisibleTo)
  
  
  • It’s “boring” legacy type we do not want in .NET Core
  
  
• There are libraries (3+ NuGet packages and ASP.NET Core app on Desktop) which will bring (indirectly) both OOB System.Net.Http 4.3.0.0 and reference to System.Net.WebRequestHandler 4.0.0.0. The combo prevents the application to run.

Solutions

1. [NOT ACCEPTABLE] Manual binding redirect – Manually per application downgrade (via binding redirects) System.Net.Http 4.3.0.0 to 4.0.0.0 – it’s difficult to understand what/where and it is per-app manual step
   
   
   
   • Note: Used today as “the workaround”
   
   
2. [CANDIDATE] Undo OOB decision – reship 4.3.1.0 as redirect to inbox Desktop 4.0.0.0.
   
   
   
   • Downside: We will break customers who depend on the new functionalities
   
   
   • Open question: Will WCF or other NuGet libraries on Desktop be affected?
   
   
3. [CANDIDATE] Sprinkle Security attributes in System.Net.Http
   
   
   
   • Do it only for Desktop (use #if), do not propagate it into other packages (compile against Desktop reference), add tests enforcing it for future.
   
   
   • Downside: Some WebRequestHandler properties specific to Desktop implementation of System.Net.Http won’t work (by design as we switched implementation)
   
   
   • Technical note: Async methods have to be SecurityTransparent (Roslyn compiler limitation), therefore they cannot call (SecurityCritical) PInvokes - for each such PInvoke call there has to be a SecuritySafeCritical wrapper method (kinda ugly, but straightforward)
   
   
   • Open question: Can we make the internal dependencies of WebRequestHandler work with new WinHttp-based System.Net.Http?
   
   
4. [REJECTED] OOB WebRequestHandler just on Desktop
   
   
   
   • Downside: Pushes the problem upstack (some APTCA assembly depends on it)
   
   
   • Downside: Some WebRequestHandler properties specific to Desktop implementation of System.Net.Http won’t work (by design) – see [3] above
   
   
   • Downside: Everyone has to add the dependency, or tell NuGet to install latest
   
   
5. [CANDIDATE] Bundle System.Net.WebRequestHandler.dll into System.Net.Http NuGet package
   
   
   
   • 2 downsides same as in [4] above:
     
     
     
     
     
     1. Downside: Pushes the problem upstack (some APTCA assembly depends on it)
     
     
     
     2. Downside: Some WebRequestHandler properties specific to Desktop implementation of System.Net.Http won’t work (by design) – see [3] above
     
     
     
   
   

[robertmclaws]

Thank you for the detailed info, we appreciate it.

What about a combination of Option 2 and re-shipping 4.3 as 5.0? Since it was a technically a breaking change, you could then ship an OOB WebRequestHandler.dll for desktop as v5.0 as well. That would let you re-implement functionality in WinHttp, deprecate properties that don't map, and gives everyone a way forward in the way that SemVer is supposed to let you. Upstream would still need to fix their code too, but thats not unexpected in a major release, and they could also just upper-bound their packages to not include 5.0.

I mean, I get the idea of wanting to ship the whole framework package groups with the same version numbers, but a) that pooch was already screwed when you guys shipped everything as 4.0 instead of following the existing Desktop versioning, and b) the actual assembly versioning is already all over the place (System.Security.Claims 4.3 package ships 4.0.2 dll, which is super-annoying when having to build binding redirects). So the damage is already done.

[dluxfordhpf]

dotnet/corefx#3 looks like the only real root-cause solution to me.

[karelz]

@robertmclaws We do not think that the version change would make a difference. Most people (app and library owners) typically just "hit upgrade to latest" on their packages, they don't care how much the version changed (minor vs. major). So the breaking impact will be exactly the same.
It is even worse that the "breaking" effect will come to light only when you combine multiple things. That's why this issue slipped through in the first place -- we do not have testing for that combo (and I would argue that's ok - one can't test all combinations, but we can discuss it in post-mortem later).

I don't think anyone cares too much about whole framework groups versions either. Honestly, if I believed it would help majority of customers, I would vote for just changing the numbers -- I just don't believe it would help almost at all. It would just change our message to "yeah, you're broken - that's what the version says, you just didn't realize what kind of things you agree to when taking the version", which is IMO lame excuse.

Given the difference in opinions, I am interested what others think: please use upvotes /downvotes on this post:

• 👍 if you agree with me, i.e. you think shipping the breaking change as 5.0 won't make a difference and most people will still be broken, confused and impacted.
• 👎 if you agree with @robertmclaws's proposal, i.e. you think shipping the breaking change as 5.0 will help people understand upfront they should rather stay away from the package, because it will break combo with another library and will prevent unnecessary pain to developers.

[dluxfordhpf]

For versioning of breaking changes, I think 5.0 is a good idea, but I don’t feel that strongly about it.

[ciel]

I'm still very confused about what is causing this problem in the first
place. You guys are really talking over my head.

I care pretty greatly about the version numbers matching, but if that has
to go in order to make this problem stop, I'll deal with it.

Didn't I read an article a few months ago about how most of the
System.Net.Http library was being sunset in favor of a rebuilt one?

On Dec 15, 2016 3:18 PM, "Karel Zikmund" [e-mail] wrote:

@robertmclaws https://github.com/robertmclaws We do not think that the
version change would make a difference. Most people (app and library
owners) typically just "hit upgrade to latest" on their packages, they
don't care how much the version changed (minor vs. major). So the breaking
impact will be exactly the same.
It is even worse that the "breaking" effect will come to light only when
you combine multiple things. That's why this issue slipped through in the
first place -- we do not have testing for that combo (and I would argue
that's ok - one can't test all combinations, but we can discuss it in
post-mortem later).

I don't think anyone cares too much about whole framework groups versions
either. Honestly, if I believed it would help majority of customers, I
would vote for just changing the numbers -- I just don't believe it would
help almost at all. It would just change our message to "yeah, you're
broken - that's what the version says, you just didn't realize what kind of
things you agree to when taking the version", which is IMO lame excuse.

Given the difference in opinions, I am interested what others think:
please use upvotes /downvotes on this post:

• 👍 if you agree with me, i.e. you think shipping the breaking change
  as 5.0 won't make a difference and most people will still be broken,
  confused and impacted.
• 👎 if you agree with @robertmclaws https://github.com/robertmclaws's
  proposal, i.e. you think shipping the breaking change as 5.0 will help
  people understand upfront they should rather stay away from the package,
  because it will break combo with another library and will prevent
  unnecessary pain to developers.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/11100#issuecomment-267446604,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAavtBRE24SlHsZHCYm5OhPbOGs6MfRzks5rIa68gaJpZM4JsdDX
.

[dluxfordhpf]

I concur. I derived some details from the email earlier today, but I still am unsure. It would be nice to see a good description of the internal problem so we understand the proposed solutions.

[chadwackerman]

@karelz I appreciate what you're trying to do here but taking a poll as to what "version 5.0" means is a total waste of everyone's time. It's GitHub socializing not engineering. I'd ship a 5.0 version TODAY that fixes this and ship a 6.0 when you figure out how to set all your little annotations right and/or refactor the code.

Statements like "Most people (app and library owners) typically just hit upgrade to latest" aren't useful. That's how Perl 6, Python 3, Rails 3 & 4, and Node.js 1,2,3,4,5 & 6 managed to splinter things. Let's not follow that lead.

[clairernovotny]

@dluxfordhpf @ciel Unfortunately, this is a hard area to explain. It's all coming from "legacy" Code Access Security that's no longer actively recommended for use.

Here's a summary of what it's purpose is/was:
https://msdn.microsoft.com/en-us/library/ee191569(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/dd233102(v=vs.110).aspx

The actual issue has to do as @karelz said with an type of one level of security transparency (by way of being in the desktop framework) trying to derive from a type that has a less restrictive security stance (because the attributes are missing in the OOB version). This is because CAS as a concept doesn't exist within anything other than desktop .net.

In the context of the docs above, look at this section on Inheritance Rules

It describes the rules required for inheritance of types across different security levels.

[dluxfordhpf]

Thanks! I’m familiar with CAS; technicals are wonderful.

[robertmclaws]

Given all of the version number changes between .NET Core, .NET Standard, et. al. in the last year (and shipping new code versioned 4.3 on NuGet that runs on 4.6.2, I understand that Microsoft might not think that version numbers matter. But as someone who singlehandedly manages a very complex application architecture, and ships over 20 OSS NuGet packages, I wholeheartedly disagree.

Hitting "Update All" without checking compatibility is the easiest way in .NET to screw up your app and not know until runtime. Adherence to SemVer is the only way to maintain any sense of sanity. Incrementing the major version communicates that something is going to be broken. When you signal that change, you then get the freedom to do whatever you want to fix it.

[gulbanana]

The proposed fix sounds good to me but I'd just like to comment on the test matrix - use of HttpClient on desktop .NET is going to be a major thing for the foreseeable future. This combo really should be brought under test, even though it's true that not every possibility can/should be tested.

[chadwackerman]

Yeah the test thing sounds like a copout to me too. Add the top 100 packages to your unit test project and make sure your crap still runs. Seems like the kind of basic engineering Microsoft used to do before realizing they could fire the testers and let "the community" sort this stuff out instead. It's just such a terrible and frustrating waste of time.

[ciel]

Because the 'old' QA that brought us Windows ME and Vista were superior.

Also, I'm sure insulting them will get the job done faster.

On Dec 16, 2016 7:46 AM, "chadwackerman" [e-mail] wrote:

Yeah the test thing sounds like a copout to me too. Add the top 100
packages to your unit test project and make sure your crap still runs.
Seems like the kind of basic engineering Microsoft used to do before
realizing they could fire the testers and let "the community" sort this
stuff out instead. It's just such a terrible and frustrating waste of time.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/11100#issuecomment-267597137,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAavtAUL3pmMiSRnFBe7DEa0y5AaZoVXks5rIpZIgaJpZM4JsdDX
.

[ciel]

Found the team leader no one invites to the holiday parties because he
treats people like shit.

On Dec 16, 2016 8:26 AM, "chadwackerman" [e-mail] wrote:

Calling out bullshit believe it or not does get the job done faster.
Otherwise you've got millionaire dev managers who haven't written a line of
code in 10 years saying "Wow this Github thing sure is neat. let's hack the
upvote/downvote emoji and hold a poll so I don't have to make a decision."
As if that's going to resolve anything after man-months of work and six
hours of triage meetings internally. It's false social signalling and
frankly the kind of b.s. engineers used to call out that brought us quality
like the Saturn V rocket. And .NET which was the best language packaging
and standard library of anything in this space long before all this online
idiocy started.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/11100#issuecomment-267604666,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAavtEigDK5EqlA4LQVlxB_lfamMgHanks5rIp9-gaJpZM4JsdDX
.

[chadwackerman]

Or maybe you found the guy who learned how to unblock his entire team by getting Microsoft to finally act on an issue that was ignored for months.

Microsoft has completely duplicate bug lists and priorities internally. Github is a bunch of social nonsense. They wouldn't take a pull request for this issue anyway so this turns into a customer support thread. The devs did a bad job here and it's okay for them to hear it.

[ciel]

There's a stark difference in the devs hearing that and being an
insufferable asshole. If insulting remarks is the only way you can convey a
point, then maybe you should stick to berating the people you're at least
paying to put up with you.

On Dec 16, 2016 8:34 AM, "chadwackerman" [e-mail] wrote:

Or maybe you found the guy who learned how to unblock his entire team by
getting Microsoft to finally act on an issue that was ignored for months.

Microsoft has completely duplicate bug lists and priorities internally.
Github is a bunch of social nonsense. They wouldn't take a pull request for
this issue anyway so this turns into a customer support thread. The devs
did a bad job here and it's okay for them to hear it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/corefx/issues/11100#issuecomment-267606461,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAavtHTdxym7pvR9KRomyIT14FVaILLFks5rIqF8gaJpZM4JsdDX
.

[chadwackerman]

I'll respond only because Microsoft uses "sort by GitHub number_of_comments" to assist in what @karelz so politely leaked as "figure out funding." You're welcome to use the GitHub emoji scale to validate your social values as well.

Dude came in and said SemVer doesn't matter. It's our duty as engineers to point out how absurd that is. It's a classic case where we need a more senior manager who has actual experience, or a more junior guy who actually knows how stuff impacts the real world. Middle managers playing mayor on GitHub is the real problem here. Now feel free to click thumbs down so we can all get on with our day, thank you.

[dluxfordhpf]

Yeah, this bug has sucked, and I am surprised to see a main path sev1 released. But IMHO the real pain from this is due to the design of NuGet: any NuGet package a project uses has to be manually referenced by all consumers of that project. That is needless duplication and bad design, it sets you up for failure. The synchronization wizard is useless when he basic design is bad. NUGET is why this bug is so painful. Otherwise we would fume, put the nasty workaround in Util, and be able to forget about it. But now we have to be careful in all 18 or so consuming projects, growing every month. THAT is why this bug is so painful – because of NuGet, the fix cannot be isolated to one project, you have to touch everything, and keep on doing it.

Also, I do echo the sentiment that Desktop/Windows .NET is what’s important. I am happy to hear about Microsoft .NET coming to Linux, but the money is on Windows, that is where we should be having the best experience, and I want that running the best.

A continual gripe my team has is “why do we have to download all these packages?” We create a console or ASP.NET project and everything it needs is on the box. You create something more modern and 5 billion downloads.

OK, I’m getting a bit far a-field. Thanks for your time and attention and please let me know if we can help you test/evaluate/check docs for typos/whatever you need we are willing to help.

[jahmai]

The reason why this issue (and dotnet/runtime#17786) have caused so much pain for us is because we moved our cloud services from Azure web roles to Service Fabric services and had to move to netcore/netstandard, but still running in full framework apps.

At first I did the binding redirect workaround which seemed to work for a time, but our devs were constantly breaking something by accidentally reverting an app.config, updating a nuget package and fighting AutoGenerateBindingRedirects (which disabling was it's own nightmare).

Finally, I solved this by forcing the use of HttpClientHandler everywhere. That involved changing our own code, raising issues in third party libraries and even using hacks like reflection and duplicating 3rd party code to work around the issue.

So whatever the solution is, if the new package doesn't support the newer HttpClient / HttpClientHandler then we won't take it. That's not a huge issue because right now things are working. However, the "real fix" needs to come soon after, since I don't want to be blocked on updating even more 3rd party packages that might be moving their code to netstandard and introducing this problem.

[karelz]

@dluxfordhpf ... It would be nice to see a good description of the internal problem so we understand the proposed solutions ...

I tried to describe the problem here: https://github.com/dotnet/corefx/issues/11100#issuecomment-267394198.
In nutshell:

• Http NuGet package 4.1/4.3 "overrides" Http 4.0 in .NET Framework, but does not have security attributes. Moreover it uses different OS component under the hood (i.e. "big change" + potentially breaking corner cases (while not directly relevant to this problem, it's a sign it bags for troubles)).
• WebRequestHandler is only part of .NET Framework, but depends on Http 4.0 and requires security attributes in Http.
• Once you take dependency on Http 4.1/4.3 from NuGet and redirect the inbox 4.0 .NET Framework version to it, you cannot use WebRequestHandler.
• To make things worse, Http has some new APIs which are part of .NET Standard 1.6 and some components depend on them. So we cannot simply roll back to old version.
• And of course the details make it even more complicated.
• Stating the obvious: There is no "right solution". It is matter of choosing the one which has less impact overall.

@chadwackerman ... taking a poll as to what "version 5.0" means is a total waste of everyone's time. It's GitHub socializing not engineering.

It's not socializing. If you look at the votes, it shows there is not unanimous opinion on the matter. So even if you don't trust CoreFX team (with 10+ years experience with .NET customers) opinion which says "people don't pay attention to major version as we wish", I hope it helps to see that even some MVPs (@onovotny) share that opinion. While other MVPs disagree (@robertmclaws).

@chadwackerman ... I'd ship a 5.0 version TODAY that fixes this and ship a 6.0 ..

It is option [2] I laid out above (with different version numbers). It is something we are exploring in parallel. Given the impact of the solution, it is not clear "yeah - that's obviously the right thing to do, let's go for it".

@robertmclaws ... Microsoft might not think that version numbers matter ... When you signal that change, you then get the freedom to do whatever you want to fix it ...

I didn't try to say we do not believe in the right versioning. We just believe that significant portion of developers doesn't care and wants everything to be fully compatible always - i.e. we cannot do whatever we want even in major version bumps. That is based on our team's years of experience with .NET servicing and shipping NuGet packages.

@gulbanana ... This combo really should be brought under test, even though it's true that not every possibility can/should be tested ...

Agreed, now that we know it is important combo we should add coverage for it. And we should poke at the area a bit more to see if we are missing other similar combos around Http NuGett package.

@chadwackerman ... Add the top 100 packages to your unit test project ...

Funny enough, that is similar to what I have proposed a year ago when we shipped bad Meta-package for UWP due to lack of the right test coverage. It just isn't that easy. To catch the interesting bugs (like this one), you have to also exercise the code. Often times you have to exercise code from both combos in the same test (not in this particular case). Overall pretty darn difficult in the test infra.
If you have suggestions and ideas we might have missed, let us know -- let's just spin off a new issue for that topic.

@chadwackerman ... and let "the community" sort this stuff out instead ...

We (I can speak for CoreFX and CLR teams) are NOT outsourcing testing of products to community via GitHub. We hold high bar on quality of what we ship.

(accidental hotkey sent the reply before I finished it ... to be continued ...)

[karelz]

@chadwackerman ... Microsoft has completely duplicate bug lists and priorities internally ...

Not true at least on CoreFX & CoreCLR teams - GitHub is primary bug database for all .NET Core. Other non-open source products ("full" .NET Framework and .NET Native) use primarily internal bug databases.

@chadwackerman ... They wouldn't take a pull request for this issue anyway ...

We would not take a pull request which is not able to address/explain all the concerns above (incl. those the person submitting it doesn't personally agree with). Are there open source projects which would take a quick hack without thinking through all its impact? Maybe if 20%+ of its customers is on the floor ... that is not the case here (yet).

@chadwackerman ... Microsoft uses "sort by GitHub number_of_comments" to assist in what @karelz so politely leaked as "figure out funding." ...

First, number_of_comments is not a metric we pay attention to (unless someone "manually" notices that it goes out of the roof). And it definitely has nothing to do with "funding".
We pay attention to anything that has impact on customers -- either number of votes, or voiced customer dissatisfaction (on GitHub, Twitter, blog posts comments, anywhere else).
This bug got on my radar as "significant customer dissatisfaction" (another MS employee on this thread raised it up as such internally), that's why I stepped in.
We are funding it as much as we can at this moment. The only higher priority would be a security issue or 20%+ of our customers is on the floor without a workaround.
BTW: Throwing in insults is not going to help speeding it up or influence the decision making.

@dluxfordhpf ... please let me know if we can help you test/evaluate/check docs for typos/whatever you need we are willing to help ...

Thanks, we appreciate your offer. We will definitely welcome help when we have bits ready (and validated on the repros we have), to validate more end-to-end scenarios. We will want to avoid a situation when we fix just some end-to-end scenarios, while leaving others broken in the same or another way.

@jahmai ... but our devs were constantly breaking something by accidentally reverting an app.config, updating a nuget package and fighting AutoGenerateBindingRedirects (which disabling was it's own nightmare). ...

Thanks! That is good example on clarifying the impact of the issue on dev daily work we need to hear and understand. It helps us understand that the combo with VS tooling makes this a true nightmare to work with. We will take it into consideration when finalizing the release plan and dates (which I will start as soon as we settle on solution).

@jahmai ... if the new package doesn't support the newer HttpClient / HttpClientHandler then we won't take it ...

Again, good feedback to hear. We are still trying to flush out the whole end-to-end fixes & release story, before we fully buy into one solution. Releasing a hack without understanding the impact on other scenarios (like yours) would be a desperate move from us -- we would consider it only, if we don't know how to fix it, or if the proper fix is extremely costly. We are not there yet.

[karelz]

Please let's keep the discussion technical from now on.

Quick update:
Solution [3] "Sprinkle Security attributes in System.Net.Http" as described above seems to be costly (6w+), so we re-pivoted its internal implementation details: We are considering to use original Http implementation from 4.0 version of the package + implement the 8 new APIs on top of it as best as we can (some may be technically breaking with workarounds). http2 support and other "new WinHttp stack" goodies won't be available by default, whoever wants them needs to call special constructor (technically breaking change, but hopefully not many folks depend on the new 4.1/4.3 under-the-hood goodies).
The cost seems to be significantly lower so far, but we are still chasing down and costing 2 lose ends (APIs), before we settle on the final plan. We will have to make some "interesting" compatibility decisions at least for 2 APIs, but they should not slow down the time to the release of the fix.

BTW: Solution [2] "Undo OOB decision" turns out to have more impact than we originally guessed (e.g. it would break .NET Standard 1.6, creating more long-term mess and explanations). We are keeping it as last resort option at this moment, if the above turns out to be too costly or otherwise unreasonable.

[gulbanana]

if you're currently looking at the edge cases around this issue, this could be worth checking out:
https://github.com/gulbanana/repro-netstandard-httpclient

this solution demonstrates that currently in vs2017rc, using netfx and netstandard results in clashing versions of system.net.http. i'm not sure how that relates to the OOB thing.

[chadwackerman]

I appreciate (and frankly admire) @karelz candor here but I'm going to sound the versioning horn one final time.

if you don't trust CoreFX team (with 10+ years experience with .NET customers) opinion which says "people don't pay attention to major version" ... based on our team's years of experience with .NET servicing and shipping NuGet packages ...

I'm not sure precisely which logical fallacy this is but you can't claim a deep nuanced experienced team understanding of versioning in the middle of a massive versioning problem like this.

Hopefully we can at least agree that NOT bumping the major version number and having a breaking change is the worst of all worlds. Yet here we are. With talk of "budget" and "6+ weeks of dev time"... to slap on some attributes. For an outdated trust system that never really worked and that I don't use. Because of neglected issues in the compiler.

Amazon Web Services shipped full .NET Core support for all their services last summer. Their recent update lowers the bar from netstandard 1.5 to 1.3. Meanwhile the Azure team can't even get blobs working.

You guys shipped bits that globally break https web requests, silently during builds and loudly at runtime, and still don't have a plan for solving it four months later. I'll drop this for now because obviously you're working on it -- but if the bigger issues here aren't keeping you up at night...

[dluxfordhpf]

Thanks so much for your candidness and your explanations.

[karelz]

@chadwackerman ... you can't claim a deep nuanced experienced team understanding of versioning in the middle of a massive versioning problem like this. ... Hopefully we can at least agree that NOT bumping the major version number and having a breaking change is the worst of all worlds. Yet here we are. ...

You're making 2 large assumptions:

1. Assumption: The breaking change is known (aka understood to be breaking) before shipping.
   
   
   
   • That's not the case for this issue -- as is common for class of "unintended breaking changes". Yeah, things sometimes slip through :(. The important metrics are IMO: fast reaction (I agree we didn't do that one right on this issue until now) and decreasing/low frequency of such mistakes.
   
   
2. Assumption: Versioning experts are capable to review every single change (which they are not) and that people in general never make mistakes (sci-fi).
   
   
   
   • In this particular case, the System.Net.Http API plan (the new 8 APIs for Desktop) was reviewed high-level before shipping also by versioning experts. They provided suggestions and guidance on options. Unfortunately, the level of breaking was not recognized upfront by anyone (neither area experts nor versioning experts), therefore the chosen solution lead to these troubles.
   
   

I also want to point out that by dismissing our experience in versioning you're basically saying "if you (.NET team) ever make a big mistake (this bug), you are not allowed to call yourself expert even on past history and customer patterns (related area)".
That's IMO big hammer and not realistic. People/teams make occasional mistakes (like this one). That doesn't make them unfit to be experts in the nearby areas (or even in the particular area). The key thing is if they can learn from their mistakes and do better in future.

@chadwackerman ... With talk of "budget" and "6+ weeks of dev time"... to slap on some attributes. ...

The cost is not from "slapping attributes", that's the easy part. The costly part is from the open question on option [3]:

https://github.com/dotnet/corefx/issues/11100#issuecomment-267394198: Open question: Can we make the internal dependencies of WebRequestHandler work with new WinHttp-based System.Net.Http?

Turns out, that's quite a lot of work, the internal dependencies are just too nasty :(

[robertmclaws]

I really appreciate all of the discussion on this. Seriously. It's awesome. Thanks for being open on this.

At the end of the day, though, here is the dotnet/corefx#1 issue: You guys rewrote a major part of the stack. The part that is central to almost everything. And you don't have even-remotely-adequate test coverage. It shouldn't have needed experts. It should have had a clearly-documented test case that was written at the time WebRequestHandler was written (or the decision was made not to port to Core), and it should have broken when you guys started monkeying with it.

After > 5 years of shipping .NET 4.0-related stuff, there really is no excuse for not having integration test coverage SOMEWHERE that would have caught this. If you find yourself making an excuse for it, you are not demanding excellence from your team.

If there had been:

• proper test coverage
• a proper QA process
• a proper beta release
• and a proper way to submit issues to specific teams responsible for specific packages

...this wouldn't have happened.

If the team:

• hadn't been so hell-bent on boiling the ocean by re-writing EVERY SINGLE ASPECT of .NET at the same time
• heeded the warnings of those of us that have been vocal about this since the ".NET 5 / MVC 6" project began

...this wouldn't have happened.

It's a combination of failures that broke .NET at-large, on a scale that I dont ever recall happening before. But you've also broken a large part of the ecosystem, and is becoming increasingly difficult to fix.

There is a _very real_ and _significant_ cost to this. We had to shut off our CI Server weeks ago and deploy manually, verifying the web.config manually every time. We deploy at least once a day. Hundreds of dollars a week in additional man-hours, not to mention the delays from having to spend that time doing something else.

Again, this is not even taking into account the countless hours dealing with existing flaws in the HttpClient architecture: that isn't disposing of instances properly, keeping TCP connections open, and caching DNS entries too long.

So, you're going to catch some flack for this. And deservedly so.

And based on how "expensive" the fix is, I hope you're putting a tiger-team of several of your best people on it to fix it right, and not putting it on one person's shoulders.

[StingyJack]

... The beatings will continue until morale improves.

Let them fix the mistake they admitted to and not be tied up in tact.

[robertmclaws]

Just so we're clear, I'm not trying to beat a dead horse with my last post. But I've seen too many excuses from Microsoft as of late. "Naming is hard." "Versioning is hard." "People make mistakes." This is the mentality of weakness, and not of a team striving for excellence. I really admire @karelz for coming out here and discussing it. But any MSFT employee needs to stop justifying what happened, and let people vent without feeling the need to spend time correcting the record. This isn't some super-edge-case exception when you use a DateTime or something. It's a colossal eff-up due to more than one person not demanding excellence in their work, and it should be treated as such. The only valid response is: "We effed up. We broke .NET. We're not going to rest until it is fixed." Because that would have been the response 5 years ago when Gu was running the show.

[chadwackerman]

I'm sympathetic to @karelz who only recently took on his position after a long time working on .NET. And he certainly shouldn't have to defend his team. I don't fault the workers here; it's obviously a management problem.

But perhaps it shouldn't be a shock to see circular logic and the kind of double-talk that's widespread at certain layers of Microsoft management not play well in front of the real-world customers who actually use the stuff.

In .NET Core Microsoft SQL Server lost the ability to write unsigned values. Again a neglected issue that's at the top of UserVoice sites going back to 2014. Enterprises do not have the luxury of changing their database schema (especially something as tweaky as unsigned) because a couple program managers can't get on the same page after three years. Meanwhile Redis and SQLite (whatever that is) work just fine, and Scott Hanselman rocks tradeshow demos as if this stuff actually works. There's definitely a widening gap between internal and external realities here and the issues need to be (politely) vented.

[dluxfordhpf]

Naming IS hard. Versioning IS hard. Microsoft has a unique perspective, where they deal with customers from all kinds of industries, ranging from the bleeding edge to the dieing vine. Concepts for development approach that seem “obvious” in gaming are foreign in CRM systems. How you approach problems, what is important and what is not important, vary. And then you add advancing technology, different approaches to problems: DAO vs ADO.NET vs LINQ vs EF to use a common easy comparison. Finally, competition in the Internet server realm, whole new hardware models with turnkey tablets, and the Balmer virus. Whole new ways of thinking and modeling problems with different limitations and advantages.

At one point I worked for a…large famous company. One product group released a product with an updated shared component that interpreted it’s call stack differently depending on how it was initialized. They never tested it with our product, and it made our product go boom every time. Obvious, but they shipped it anyway. We called this the “friendly fire” incident.

I missed a bug once where if we installed to the D drive, on uninstall under certain conditions the product could…delete all files on the HD. We had to burn $40k of pressed CDs.

People aren’t perfect and we do make mistakes. Blame, however, is a useless concept. It makes us feel powerful through destructive concepts like shame and humiliation. Humility and forgiveness are more powerful. That’s how we do better, learn better ways, and sometimes simply why something is important in the first place.

Yes the problem makes me angry too, but it is getting fixed. And you know what – open source problems like this languish for decades before anyway bothers to even try. Just try to Kerberos a Linux system or look into GSSAPI. InitializeSecurityContext/AcceptSecurityContext is SO much easier. Money matters.

[robertmclaws]

At one point I worked for a…large famous company. One product group released a product with an updated shared component that interpreted it’s call stack differently depending on how it was initialized. They never tested it with our product, and it made our product go boom every time. Obvious, but they shipped it anyway. We called this the “friendly fire” incident.

Those were two different products. .NET is a single product. And I'm not trying to blame any one person. I blame the process. It feels like because Microsoft went OSS, they threw the processes that helped them ship the most stable development framework in history out the window. I feel like this would have been caught if it had shipped in .NET 4.6.3 instead of on NuGet.

[chadwackerman]

The clumsy transition to OSS certainly isn't helping matters. I don't blame OSS but Microsoft certainly seems to be proud to make all the obvious mistakes.

I use words like "quality" to describe software but they now use words like "engagement." Running the instrumented compiler that phones home an extra twenty times a day because I have to manually edit .config files is not "additional engagement." But this is the myth of the entire internet these days.

Consider the BashOnWindows team that decided to have a bunch of people with zero Linux experience write a Linux usermode shim live on GitHub. The work is a basic but tedious check off the boxes engineering exercise and there's absolutely no reason to involve community feedback for feature prioritization. But away they went.

Six months into it "the community" had to tell them it couldn't handle files that end in a period. This is not software engineering; it's some sort of weird marketing exercise. Managers who are implicit in the scheme deserve Feedback. And some of it may be unpleasant to hear.

Edit to add: similar deal with the Bash debacle. Shipping broken bits live, weird ties to other components (could only be installed as part of Windows Update), and of course Scott Hanselman somehow doing live demos despite the fact that it couldn't run tar let alone a compiler.

[jahmai]

Some of the rhetoric here smacks of "the good old days" of Microsoft, which is ironic because all the changes we're witnessing now are a direct response to customer demand for Microsoft to evolve.

We were sick of closed source and draconian reverse engineering (de-compiling) rules, so we got references sources and now open source.

We were sick of .net Framework bugs that affected every application installed on a machine and took a long time to fix and required enterprise distribution and IT departments to qualify every .net Framework version update before it could happen, so now we're moving toward shipping nuget packages with our apps so we can push the fix as soon as it is available.

We were sick of Microsoft Connect shutting down every other bug as "working as designed" or taking 6 months to even schedule for a release, so now we have Github projects managed much more closely by the team who actually writes the code, and the whole community can easily give their 2c on every community raised work item.

We were sick of Microsoft taking every good idea in the community and making a proprietary clone that didn't play well with non-Microsoft tools, so now we get to use NPM, Gulp, NodeJS, etc

We were sick of C# only being viable for Windows, and projects like Mono struggling to even keep par, while having to work around bugs or lack of functionality with #ifdef everywhere, so now we have Xamarin propelling Mono development with reference sources and allowing us to code latest C# language and share the same code base on .net, UWP, iOS, Android and netcore.

All of this is happening at once because it has. We're not going to wait for Microsoft to play catch up while it ships perfect stable pieces of functionality that only solves half our problems at any given time.

My issue isn't that all this change is happening at once. Nor that there are issues like this making an appearance. The real issue for me is that _it took months for this issue to even be recognised as a major issue_ and it's taking _even more months to figure out how to fix it_.

[chadwackerman]

Three weeks and no update. Happy New Year everyone.

Edit to add this quote from @karelz since understatement seems to trigger a slightly different group of special snowflakes here:

I will post updates (hopefully with more technical details) when we have them, at least on weekly basis.

[pollax]

@chadwackerman seriously, do you understand that your bad attitude won't help this issue getting fixed faster? And apart from that, you are making a fool out of yourself here. Please improve your tone.

[robertmclaws]

What tone? He made a statement, which has the benefit of being accurate, then wished everyone a Happy New Year. If you took it negatively, that's your issue.

And regarding that first statement: this is a major breaking issue that doesn't show up until runtime. Five years ago ScottGu or Rob Howard would have had a team on this 24-7 till a fix shipped. Now it's just "you know, we'll get around to it."

You know what will make people happy? Fix the problem. Otherwise, there are some pissed customers out there, and some of them are on this thread. They have every right to be pissed. So find something else to do with your indignation, @pollax. You have not contributed anything meaningful to this thread, and no one anointed you the GitHub Thought Police. You also haven't wasted > $5K of your company's money on this problem.

[pollax]

Ok, maybe I read to much into it and because of the tone he used in previous comments, I read this last one with a slightly ironic tone. Sorry about that.
Regarding the issue; I hear you. I'm troubled by it myself (otherwise I wouldn't be watching this issue so closely) so please don't make assumptions about what I have/haven't wasted in terms of money/resources on this. But as a developer, I know that the worst thing is to have someone staring at you when trying to fix a major issue.

[dluxfordhpf]

I concur with your read of sarcasm and agree, it is not productive. I also concur with the feeling of “when is it gonna be fixed,” though. Unfortunately this main-path CAT1 issue just didn’t attention until many people got quite vocal about it. I hope there is some process improvement internally to resolve this; it would suck for us all if the only way we can get fixes is via curse words.

[niemyjski]

I'm also getting a report of this happening on full framework with our Exceptionless nuget package. Any solid fixes or work arounds?

[annemartijn0]

I also get this self hosting ASP.NET Core from a Windows Service running full framework 4.6.2. dependency NETStandard.Library 1.6.1 forces me to use System.Net.Http 4.3.0.

[enyim]

dependency NETStandard.Library 1.6.1 forces me to use System.Net.Http 4.3.0.

and this is my biggest problem with this whole situation

more and more packages are taking a dependency on the meta-package, forcing me to upgrade every dependency I have (or fight an uphill battle of downgrading/ignoring specific ones)

this goes contrary the previous promise of "mix and match" of system dependencies

I had to downgrade my system to 4.5.2 from 4.6.1 (& losing significant time), because every second package brought in .net 1.6 and its buggy System.Net.Http

If these were only 3rd party packages, well, so be it, I can go around them, pester the maintainers to use more granular dependencies, but most of my deps were from MS, and I'm expecting MS to use granular dependencies, not just slap .net 1.6 into the nuget file

[karelz]

I planned to send an update for last few days, so here it is:

@CIPop's work was preempted by a security issue. His work was picked up by @davidsh. @davidsh plans to submit PR early this week (i.e. any day now).

Here are details on our execution plan & status:

High-level plan:
A. Replace WinHttpHandler with HttpClientHandler in net46 build of CoreFX
B. Implement 8 new APIs on HttpClientHandler we introduced in 4.1.0.0 OOB package

Execution plan:

EDIT 2017/1/12 - the execution plan was moved to the top-most post https://github.com/dotnet/corefx/issues/11100#issue-173058293

[karelz]

@niemyjski Any solid fixes or work arounds?

There's a workaround to downgrade the offending package to 4.0.0.0 (see https://github.com/dotnet/corefx/issues/11100#issuecomment-246469893), unfortunately per comments above, any edit to NuGet dependencies will bring it back up -- which is the key reason why this issue is so painful.

I hope there is some process improvement internally to resolve this; it would suck for us all if the only way we can get fixes is via curse words.

I have some ideas (and questions) about the process improvements here, to prevent future important impactful issues to go unnoticed for so long. However, I am intentionally deferring that discussion AFTER we have a fix shipped.

[dluxfordhpf]

Woot!!!

[georgiosd]

Well done guys @karelz

[gulbanana]

Thanks for the update. PlatformNotSupported would be better than nothing since they're new apis that legacy software doesn't rely upon.

[gulbanana]

It sounds like post-fix, unification through binding redirects will still be required, but a redirect to the newer rather than the older assembly, which nuget can generate correctly?

[davidsh]

Now that PR dotnet/corefx#15036 is checked in, this problem should go away w.r.t. net46. System.Net.Http.dll is now using built-in HTTP stack from .NET Framework. This means it has 100% compatibility with WebRequestHandler.

Please try out the latest packages on the MyGet dev feed. You'll want to use the latest built System.Net.Http.dll packages which as of today is:
https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

version 4.4.0-beta-24913-02.

You can use dev feed packages via changing your NuGet package source feeds either with command line tools or Visual Studio.

Instructions:

Command-Line:
Add the following in nuget.config, under element:

<add key="myget.org dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />

Visual Studio:
In VS, Tools->Options->Nuget Package Manager->Package Sources -> Add, under Source, use this url, https://dotnet.myget.org/F/dotnet-core/api/v3/index.json

In either case, make sure you list the MyGet dev feed as the first feed in the list.

[davidsh]

And to summarize the fix in dotnet/corefx#15036, we ended up with 100% app-compat with the original .NET Framework System.Net.Http 4.0 API surface and 100% app-compat when using WebRequestHandler.

In terms of the level of support for the newer properties added to HttpClientHandler (that are part of .NET Core 1.1 and beyond), the following summarizes the expected behavior of these newer properties when running on the 'net46' target:

1) public bool CheckCertificateRevocationList
Throws PlatformNotSupportedException

2) public X509CertificateCollection ClientCertificates
Implemented

3) public ICredentials DefaultProxyCredentials
Implemented

4) public int MaxConnectionsPerServer
Implemented against the current ServicePoint logic

5) public int MaxResponseHeadersLength
Implemented

6) public IDictionary Properties
Implemented

7) public Func ServerCertificateCustomValidationCallback
Implemented except that first parameter is always null due to current inability to map internal HttpWebRequest to HttpRequestMessage

8) public SslProtocols SslProtocols
Throws PlatformNotSupportedException

[dluxfordhpf]

Woo hoo! Thanks guys!

[karelz]

Did anyone get a chance to validate @davidsh's bits? We would really welcome some end-to-end validation on the 7-ish scenarios that were hinted on this thread.

We were able to validate @onovotny's simplified repro so far. Would be great to get couple of more confirmations on repros we do not have locally. Thanks!

Once we have that, we will move towards porting the change into 1.1 branch and release a patch -- hopefully next week.

[dluxfordhpf]

I will get some test done this week. I just got pulled into an escalation so I can’t start for 1-2 days, at a guess.

[karelz]

@dluxfordhpf thanks! Appreciate your help.

[annemartijn0]

I just installed 4.4.0-beta-24913-02 into my project. It seems to help my case.

Case: Self hosting ASP.NET Core from a Windows Service running full framework 4.6.2. dependency NETStandard.Library 1.6.1 forces me to use System.Net.Http 4.3.0.

The myget feed is very slow in my experience. It took a few minutes to install System.Net.Http 4.4.0-beta-24913-02
OK https://dotnet.myget.org/F/dotnet-core/api/v3/registration1/system.net.http/index.json 82079ms
OK https://dotnet.myget.org/F/dotnet-core/api/v3/registration1/system.net.http/index.json 93813ms

[karelz]

Thank you @annemartijn0 for confirmation and for describing the scenario!

[chadwackerman]

It is currently taking five to seven minutes for this page to return a result:
https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

Why is Microsoft outsourcing vital infrastructure to these shady little companies and stupid little sites? You're really choosing to pin your business AND my business on a company in Belgium called Tech Tomato?

Anyway... I already had a myget feed but it wasn't pulling down this library until after I restarted Visual Studio, clicked the "update" button buried in a dialog somewhere, then restarted Visual Studio a second time. I'm currently typing this while Visual Studio 2015 attempts to restore my packages.

Update: Visual Studio is still churning but my refresh of the myget page finally appeared. It shows perhaps a dozen downloads a day of this version of the library. Meanwhile Visual Studio summarizes it as "System.Net.Http - 75.8K downloads." I've always assumed these stats were telling me the wrong thing but here's a great example of why it's not what I want. At a minimum please tell me the current version status versus the summary so I don't unintentionally become an alpha tester without explicitly opting into the madness during this absurd moment in .NET history.

[chadwackerman]

5 hours later and I'm still unable to try this patch:

Attempting to gather dependency information for package 'System.Net.Http.4.4.0-beta-24913-02'
with respect to project '...', targeting '.NETFramework,Version=v4.6.1'

Five minutes later...

Package 'System.Net.Http 4.4.0-beta-24913-02' is not found in the following primary source(s):
'https://dotnet.myget.org/F/dotnet-core/api/v3/index.json'. Please verify all your online package
sources are available (OR) package id, version are specified correctly.

.NET is a tire fire.

[PureKrome]

Easy-up there @chadwackerman . You've _opted in_ to their beta/alpha/non production feed so this means you're ready to except problems, etc. That said, I've been using myget for ages now (paid subscription) and have very very little issues with it. So maybe .. just maybe .. it might be your computer/net-connection/whatever? (I've had some real great wins using MyGet over the last 12+ months).

.NET isn't a tire fire, right now. Sure, there's _heaps_ of cheese getting moved around and lots of moving parts and stuff which can (and do) go wrong. But there's heaps of people doing great stuff with the RC and RTM bits released. I've personally decided to use public release stuff only - not even BETA's or RC's anymore. So my expectations are: less problems but have to wait longer. So if you're finding some of this stuff really frustrating for you - maybe just wait a bit longer for some of this dev work to hit RTM status.

The work the corefx team (and the others in the .NET-core world) have been great and an excellent welcome with respect to going on github/open-and-transparent, compared to the bad-ole-Balmer-day's, so lets all try and stay positive and helpful to the repo-maintainers. Everyone here is human and all just trying to make the world a better place : one byte at a time.

[jahmai]

@chadwackerman It looks like the myget feed suffering right now. I'm not 100% sure how myget works, but I think if you have a dedicated host ("dotnet.myget.org") then the feeds are actually tenanted and have QOS limits.

Going here shows the package exists, but it takes a while to come up: https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

[chadwackerman]

@PureKrome I'm an ex-Microsoft dev manager who was asked to check this build (see above). After personally escalating this bug internally because nobody in the .NET team was even aware of it. And now they can't manage to ship me a damn binary.

I know a tire fire when I see one. I used to put them out for Microsoft for a living.

[SvenAelterman]

@chadwackerman I can understand your frustration with the issues with the feed. However, the name calling ('stupid little sites' and 'shady little companies') is out of order.
Tech Tomato is founded/run by qualified individuals: @maartenba, MVP awardee by your former employer, and @xavierdecoster, current MS employee; in a country (which I am partial to) that encourages innovation. The name of the company is hardly relevant and that the company was founded in Belgium shows that the .NET Core team looks beyond Redmond, WA for solutions.
I am looking forward to your constructive contributions to help resolve this issue.

[chadwackerman]

Content edited by moderator

A Code of Conduct report was submitted against some of the comments in this thread and as a result we have removed some material judged to have violated that code. If you have any questions or concerns about that then please email [email protected].

[Kwal]

@chadwackerman your former position in the company does not entitle you to bash anyone or engage in name calling. In addition, the vast majority of your 'contributions' to this thread have not only been unconstructive but petty, childish and off topic. Please troll elsewhere.

[martinwoodward]

Hey folks - Martin from the .NET Foundation here, wanted to explain this.

I fully appreciate that the original issue raised by this thread is frustrating and that people want it fixed. I also understand some of the wider concerns around quality and delivery and the strength of feeling in general. The team is working on resolving this issue, they’ll continue to keep folks updated.

In the meantime, please keep the tone professional and refrain from making comments that could be perceived as personal attacks against others. I've been deliberately light on the edits made as I wanted to preserve some of the strength of feeling and not overly sanitize things, but please keep things civil.

[chadwackerman]

Has anyone else tried to contact Tech Tomato? No response to my inquiries.

Here's what it looks like if you try to do actual work sorting out weird build warnings after adding the feed:

Attempting to gather dependency information for package
'System.Net.Http.4.4.0-beta-24913-02' targeting '.NETFramework,Version=v4.6.1'
Gathering dependency information took 1.55 min

Attempting to gather dependency information for package 
'Microsoft.Extensions.Configuration.Json.1.1.0' targeting '.NETFramework,Version=v4.6.1' 
Gathering dependency information took 2.2 min

... and so on.

And this link still takes 5+ minutes to show a page:
https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

That this is a) not fixed yet b) ignored by the community and c) tolerated by the .NET team as part of day to day development serves to reinforce the obvious flaws in process, tooling, culture and management that led to this bug to begin with -- plus the bug itself being ignored for months until I personally escalated it internally.

I'm looking forward to the post-mortem @karelz promised.

[maartenba]

Hi Chad,

That page load time is something we are looking at. The NuGet restore time is not normal. If possible can you reach out to us (MyGet) via support at MyGet.org with mention of your NuGet.exe version in use as well as a trace with the -Verbosity Detailed switch enabled? This will definitely help us in pinpointing any perf issues.

Thanks!

[chadwackerman]

Package Manager Console Host Version 3.5.0.1996

I'll look at getting logs from the command line as Visual Studio goes from rock solid to unstable once I add the myget.org feed. And once an error occurs pulling down a package the whole thing tips over.

p.s. @karelz: tire fire.

[maartenba]

Can you try this command line using the latest (nightly) NuGet.exe from www.nuget.org/downloads?

Exact command: NuGet.exe install System.Net.Http -Version 4.4.0-beta-24913-02 -Verbosity Detailed

This should output all intermediate download actions as well. Thanks!

[chadwackerman]

@maartenba:

On the link you sent, "latest" points to https://dist.nuget.org/win-x86-commandline/latest/nuget.exe. That is the version I already have, so I don't think that is a nightly.

I downloaded the nightly package NuGet.CommandLine.4.0.0-rtm-2254.nupkg, ran nuget.exe install on it, and it failed trying to pull down files from myget.org. FYI: 1.5 seconds to return a 404 from dotnet.myget.org.

If this is not the correct way to install a nightly build or local package please inform. You can email me at this username at gmail if you prefer to take this offline.

Still happy to help but... wow. Things really should be moving in the direction of simplicity, not me troubleshooting the secondary package host for the primary package host using a nightly drop of the package manager that, despite being on version 4.0.0-rtm, somehow has five distinct manual methods for updating itself on its distribution page, each of which requires manual user intervention.

p.s. @karelz... oh, never mind. :)

[maartenba]

Completely agree, but logs would be very helpful in finding out where the bottleneck lies (whether secondary package host, the client itself, a crazy CDN node, cosmic rays, ...). Will contact you offline to see if we can figure this out. Really appreciate your help in this.

[karelz]

I have noticed myself that myget.org is pretty slow, but I was able to install the package in question successfully in "reasonable" time (on public Wi-Fi network).

OK https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.4.0-beta-24913-02 971ms
Installing System.Net.Http 4.4.0-beta-24913-02.

We should definitely look into the overall myget.org slowness, but probably outside of the scope of this issue -- it is not typical customer scenario after all. @maartenba where's the best place to discuss that?
Over here (on this particular issue), let's focus how to unblock the end-to-end validation, e.g. by using some creative workarounds.
Also I wonder if other folks have been blocked so badly as @chadwackerman, or if @chadwackerman's experience is extremely bad, compared to others.

@chadwackerman given that the main goal here is to validate the end-to-end scenario, I wonder if you can provide your scenario steps? Someone else here (who is not so badly blocked by myget.org usage as you are) could finish the validation in such case. That should lower your pain and wasted time on your side. Of course, assuming it is doable and doable cheaply-ish from your side.

The last resort alternative, which I'd like to avoid, is to skip your scenario validation entirely -- if we get few other end-to-end scenarios validated, we might be ok-ish (assuming the worst case scenario where @maartenba won't be able to troubleshoot your super-bad myget.org experience :().

[maartenba]

@karelz taking this offline with Chad, will report back here once we figure out what's going on.

[karelz]

Background facts:

• myget.org feed is used by daily builds, CI, dev-workflow, etc. So it gets hammered a lot on daily basis. None of these perf problems manifest in those scenarios (when they did in the past we acted on them, because they affected our devs' workflow - we brought 30-60 mins sync times to <5 mins in last few months). My understanding is that it is rare for anyone on .NET team or in community to use myget feed in VS - which IMO explains why the bad perf went unnoticed in that scenario.
• This issue was escalated to me by a coworker on this thread on 2016/12/9 - that's why I joined the discussion here. I am not aware of any other internal escalation. Given that I continuously push this issue forward internally across multiple teams (almost on daily basis), I think I am aware of all events related to this issue.

[chadwackerman]

@karelz Yes, I escalated this through internal contacts on that same date 2016/12/9. I'll provide details, emails and the projects I work on privately after your postmortem and we can wrap this up offline. Perhaps in person, perhaps using alcohol.

I since verified that MyGet.org is misbehaving from multiple friend's machines on both the East and West Coasts. Mine is a recent clean install of Visual Studio 2015, no add-ins, definitely no ReSharper. The user feedback and error handling in Visual Studio is poor. Even here other users report similar lags. I'll drop this for now to free up the thread but let's not pretend that MyGet isn't borked and the whole NuGet packaging system (and NuGet's ability to update itself) doesn't faintly smell of burning rubber and isn't also a contributing factor to this bug. Both originally as part of the root cause and even today as part of trying to test and ship the fix.

[maartenba]

Not sure if we should keep discussing this in this thread or open another thread for it. Anyway: status report!

We're in touch with @chadwackerman via e-mail - thanks Chad for the log you sent!

On the "slowness" - preliminary profiling tells us that this is caused by the number of package versions and the impact of that fact on download size for protocol data. For example, some packages require 4 MB of JSON data to be downloaded and parsed to gather metadata. This causes a big memory usage spike in the NuGet client and the need for parsing a boat load of JSON data - definitely room for improvement there although the 4.0.0 nightlies of the client seem to behave better.

On MyGet side we'll be implementing paging for these protocol blobs to reduce download size on the protocol. May take a week for this to go live. We'll also be profiling both server and client on these requests and see if there is room for optimization on either side.

We're also looking at speeding up that page load time (but that is secondary, having install/restore work fast is priority).

[chadwackerman]

After hours of experimentation I was able to upgrade to System.Net.Http 4.4.0-beta-24913-01 without crashing Visual Studio. I then tried to upgrade from 24913-01 to 24913-02 and got a proper error instead of a crash.

It's 2017 and updating the nightly build of the core HTTP component for the entire system from "Monday's build" to "Tuesday's build" takes more than five minutes of wall clock time on an 8-Core i7 and requires more than 4GB of RAM.

The project in question is a webjob (a rebranded command line app) consisting of 27 lines of code.

Interestingly @karelz states that this works great for the whole .NET team and he has no problems at all sucking down the binary even while casually sipping a latte from the local Starbucks.

I was able to install the package in question successfully in "reasonable" time (on public Wi-Fi network).
https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.4.0-beta-24913-02 971ms

Here's some alternative facts:

Attempting to gather dependency information for package 'System.Net.Http.4.4.0-beta-24913-02' with respect to project 'webjob', targeting '.NETFramework,Version=v4.6.1'
Gathering dependency information took 4.22 min
Attempting to resolve dependencies for package 'System.Net.Http.4.4.0-beta-24913-02' with DependencyBehavior 'Lowest'
Resolving actions to install package 'System.Net.Http.4.4.0-beta-24913-02'
Resolved actions to install package 'System.Net.Http.4.4.0-beta-24913-02'
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Newtonsoft.Json.Linq.JContainer.ReadContentFrom(JsonReader r)
at Newtonsoft.Json.Linq.JContainer.ReadTokenFrom(JsonReader reader)
at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader)
at NuGet.Protocol.HttpSource.<>c.b__15_0(Stream stream)

[richlander]

Ya, can you guys please move this MyGet/NuGet problem to a new issue? It has no relationship to the original issue.

@onovotny - You might consider re-opening your issue to. This issue is now super long, so the usability of the issue has been reduced.

[chadwackerman]

@richlander instead of policing the thread would you instead try to install the fix and report back if it solves your specific issue? Everybody seems to be blocked by this important issue yet people would rather play GitHub ping pong than do the actual work to test the fix.

[annemartijn0]

Test all ~7 end-to-end scenarios reported by community (ask for help from community, provide steps to consume master packages on myget)

What are the 7 scenarios? Can everyone help?

[clairernovotny]

@richlander I'd be happy to re-open an issue, are you just looking for me to clone this one with @karelz's updates? Not exactly sure what the ask is.

[karelkrivanek]

My usecase was working with the Azure Storage API, which was last working in 4.0.1-rc2-24027. It looks like this is working now. I've only done a quick test, since it took about 20 minutes to update all the packages in my projects from MyGet.

[chadwackerman]

@onovotny Don't reopen it. We've got momentum here and we are inches away from at least a partial resolution. The thread is overwhelming for anyone just joining now, but like any other long-running unresolved issue that reveals deeper issues, it has to be. GitHub sucks for things like this.

I verified that the new binary works compatibly on one local app. No surprise there. I then tried to cut and paste those dependencies to my webjob project and I couldn't get it to work on Azure. Webjob loads fine but it fails when triggered, unable to load System.Net.Http. This is obviously my fault and I know the drill to fix it. But I'm pretty much back where I was when this bug first opened: tweaky binding remaps and when I touch NuGet all hell breaks loose, I waste an enormous amount of time, and my project passes all tests locally yet breaks at runtime upon deploy.

[jahmai]

So our scenarios are:

1. A dependant (Raven.Database) package uses WebRequestHandler, which was breaking at runtime as reported in this issue.
2. Our code uses the new HttpClientHandler type and properties.

Previously I tried the binding redirect fixes but that lead to strife, I then ended up using hacks (reflection injection, duplicating and modifying 3rd party code) to work around the issues.

I have updated to the beta and removed all the hack code, and everything (our application, end-to-end, from the browser to the database) appears to be working locally! I haven't yet tried the code on the Azure platform so I'll confirm back once that is done, but I consider this significant progress.

Also, when updating the packages I found it tolerable to use the Visual Studio Package Manager Console and use this command to update the packages (rather than add another feed config and then use the UI, which is incredibly painful):

Update-Package System.Net.Http -Version 4.4.0-beta-24913-02 -Source https://dotnet.myget.org/F/dotnet-core

This took 6 minutes 53 seconds for 20 projects to be updated.

Note that all our projects had the following binding redirect automatically generated, I did not need to fiddle with binding redirects at all:

    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.2.0.0" />
      </dependentAssembly>
    </assemblyBinding>

Almost time for high-fives!

[chadwackerman]

@jahmai When I used that command line it did not update my reference from 4.0 to 4.2 nor add the necessary copy flags. Make sure those are set for System.Net.Http and dependencies and it should work fine on Azure.

[pollax]

Our scenario is a direct dependency on types in System.Net.Http.
I tested Update-Package System.Net.Http -Version 4.4.0-beta-24913-02 -Source https://dotnet.myget.org/F/dotnet-core on one project and so far it seems to work well. That's very good news.
Update Forgot to mention that binding redirects from 4.0.0.0 to 4.2.0.0 were applied automatically.

Regarding the Nuget/MyGet issues, I got the following output for this single project:

Gathering dependency information took 37.47 sec
Executing nuget actions took 35.15 sec
Time Elapsed: 00:01:14.7537429

Note that I'm in timezone UTC +01:00, don't know when MyGet is getting the most traffic.

[maartenba]

@pollax Thanks. We found the issue (see my last comment above) - client+server combination. Working on making it better.

[MikeGoldsmith]

I can confirm using the beta System.Net.Http library from MyGet worked for my scenario:

• .NET 4.6 console application with a dependency on a library that uses System.Net.Http

It took around 90 seconds for the nuget package to download from MyGet and the bindingRedirect in the app.config was applied correctly.

I'm happy to help test more scenarios if they have been described somewhere.

[chadwackerman]

Interesting side effect: adding the 4.4.0-beta of a Windows-only .NET library broke deployment of a .NET Core application on Linux.

"dotnet restore" is hardcoded to 60 second package retrieval times. And there's no flag to select just a specific platform like "dotnet publish" supports. So for a cross-platform library your tiny Linux worker node downloads a bunch of Windows binaries unnecessarily -- then times out and fails when it hits MyGet. Amusingly the out of memory problem that crashes Visual Studio on a 32GB machine doesn't affect a 0.75 GB Linux worker because it swaps to death instead.

Yes, I logged this elsewhere. And yes, it does relate to this bug even if you don't see it yet.

[karelz]

Thanks everyone helping us verify the scenarios! We got so far 5 confirmations - see detailed list with links in top-most post. I consider it good enough validation to move to next stages.

• [4.a.ii] I am still chasing down the NuGet analysis.
• [5.a] I will ask @davidsh to prepare PR against release/1.1.0 branch.

• [5.b] I am prepping the process for release (we need director-level approval, but given the impact on customers I don't expect any push back).

• I am preparing official info about the patch release - listing all technical breaking changes and workarounds (thanks to @davidsh for the data).

If anyone discovers any concern in the meantime (related to this specific issue), please say so ASAP. Fingers crossed.

[richlander]

@onovotny - It does look the issue is now progressing forward, so let's just continue with this issue. It's great to see that folks are making positive progress.

[chadwackerman]

@karelz You can check off my scenarios which include 4.6 command line (Azure webjob) with ServiceBus dependency and a 4.6 app with dependencies on Azure Batch. Also a third app with dependencies on AWS and Dropbox which I previously moved to .NET Core just to get away from this issue (I pulled an old version to test today).

dotnet restore problem revived at https://github.com/NuGet/Home/issues/2534, Contributor Covenant fixed backchannel, tires kicked at https://github.com/Azure/azure-storage-net/issues/372, MyGet logs sent, additional performance logs requested at https://github.com/dotnet/cli/issues/5328, and I bought a huge bag of popcorn for the postmortem discussion.

[karelz]

Thank you @chadwackerman for your help and confirmation! Sorry for the trouble you encountered en route.
I have updated the top-most post.

[karelz]

From my list above:

I am preparing official info about the patch release - listing all technical breaking changes and workarounds (thanks to @davidsh for the data).

I have added info to the top-most post - see section "Impact of the change - Breaking changes" for list of technical breaking changes (4), each with a workaround.

[karelz]

NuGet libraries impacted by the technical breaking change (described in top-most post) -- luckily "just" 4 NuGet libraries which use any of these new APIs:

System.Private.ServiceModel_4.3.0

• https://www.nuget.org/packages/System.Private.ServiceModel/
• Author: dotnetframework
• Downloads: 11K (latest version) / 800K (total)
• Description: Internal implementation package not meant for direct consumption. Please do not reference directly. Provides implementation of System.ServiceModel packages.
• Notes:
• Status: Package not affected - @zhenlan @mconnew from WCF team confirmed they use the properties only in .NET Core builds. On Desktop they fall back to built-in Desktop binaries.

Consul_0.7.2.1

• https://www.nuget.org/packages/Consul/
• Name: Consul
• Author: https://www.nuget.org/profiles/highlyunavailable
• Downloads: 500 (latest version) / 65K (total)
• Description: https://github.com/PlayFab/consuldotnet/blob/master/CHANGELOG.md
• Notes:
  
  
  
  • 2 usages: https://github.com/PlayFab/consuldotnet/search?utf8=%E2%9C%93&q=ServerCertificateCustomValidationCallback
  
  
  • Issue filed on GitHub: https://github.com/PlayFab/consuldotnet/issues/79
  
  
  • Email sent to author via nuget.org
  
  
• Status: Package not affected (GitHub issue https://github.com/PlayFab/consuldotnet/issues/79 closed)

Octopus.Client_4.6.1

• https://www.nuget.org/packages/Octopus.Client/
• Author:
  
  
  
  • https://www.nuget.org/profiles/paulstovell
  
  
  • https://www.nuget.org/profiles/OctopusDeploy
  
  
• Downloads: 250 (latest version) / 55K (total)
• Description: None
• Notes:
  
  
  
  • Issue filed on GitHub: https://github.com/OctopusDeploy/OctopusClients/issues/121
  
  
  • Email sent to support alias: [email protected]
  
  
  • Email sent to author(s) via nugget.org
  
  
• Status: Package not affected (details in GitHub issue https://github.com/OctopusDeploy/OctopusClients/issues/121)

Geotab.Checkmate.ObjectModel_5.7.1701

• https://www.nuget.org/packages/Geotab.Checkmate.ObjectModel/
• Name: Geotab SDK
• Author: https://www.nuget.org/profiles/neilcawse
• Downloads: 150 (latest version) / 10K (total)
• Description:
  
  
  
  • SDK library for accessing MyGeotab customer databases
  
  
  • See a full list changes at https://my.geotab.com/sdk/default.html#/whatsNew
  
  
• Notes:
  
  
  
  • Email sent to author (geotab.com CEO) via nugget.org
  
  
• Status: Owner contacted, waiting on response

Sorry for the inconvenience to all affected packages authors.

[maartenba]

On the Visual Studio / NuGet crash / swapping on Linux: the reason for this is in the way the NuGet protocol works. I have documented findings in this issue: https://github.com/NuGet/Home/issues/4448

On MyGet side we will be deploying a change after the weekend (currently in test, ETA in production Monday Feb 7) that alleviates this server-side.

[maartenba]

Fix on MyGet side is live. Should work fine in Visual Studio. When using NuGet.exe, make sure to use the NuGet.exe embedded in https://dotnet.myget.org/F/nuget-build/api/v2/package/NuGet.CommandLine (4.0 nightly) - the 3.5 seems to fail figuring out dependencies (but not always). Bug logged: https://github.com/NuGet/Home/issues/4512

[chadwackerman]

Thanks for the deep dive on this @maartenba. Never underestimate the impact even a minor tooling fix can have!

Interesting that the entire .NET team could miss both the Visual Studio crash and the NuGet problem.

I once asked an room of 80+ Microsoft devs to raise their hand if anyone was having problems setting breakpoints in the debugger. I got two hands. Compiler changed symbol format, you couldn't build the project without the latest compiler, but the debuggers hadn't updated yet.

For months nobody could set a breakpoint. On two platforms you couldn't even get a stack trace! I get called into the build lab at 1am because I'm the only other person around, get a screen full of assembly for a processor I've never even seen docs for, pull up a backtrace and the debugger crashes into the debugger.

When you change the project format while you're changing the code that parses the project format while you're dogfooding a new version of the package manager that plugs into the new version of Visual Studio -- stuff like this results. A mortal can only deal with so much change at once and this is why devs keep dropping the ball everywhere. Both us and them!

[jirisykora83]

If somebody want simple powershell script for fix bindingRedirect in all app.config here it is. Probably it is not best solution but right now i have project with lot of webjobs sub-projects and it is really frustrating manually changing all file binding after updating some package.

param(
    [string]$SourceDirectory,
    [string]$Package,
    [string]$OldVersion,
    [string]$NewVersion
)

Write-Host "Start fixing all app.config"

[array]$files = get-childitem $sourceDirectory -Include app.config App.config -Recurse | select -expand FullName
foreach ($file in $files)
{
    Write-Host $file
    $xml = [xml](Get-Content $file)
    $daNodes = $xml.configuration.runtime.assemblyBinding.dependentAssembly
    foreach($node in $daNodes)
    {
        if($node.assemblyIdentity.name -eq $package)
        {
            $updateNode = $node.bindingRedirect
            $updateNode.oldVersion = $OldVersion
            $updateNode.newVersion =$NewVersion
        }
    }
    $xml.Save($file)
}

Write-Host "Done"`

Example use:
./scripts/FixAppConfig.ps1 -SourceDirectory "--project-path--" -Package "System.Net.Http" -OldVersion "0.0.0.0-4.3.0.0" -NewVersion "4.0.0.0"

[georgiosd]

When does this go public again? :)

[karelz]

Our change got into release/1.1.0 branch on Tuesday - package version 4.3.1. All packages in the branch were fliped to stable yesterday (independent effort, but helps us as well :)).
@davidsh will do sanity test on myget feed (ETA: today), then we will ask for one last validation here by community (ETA: today, EOD). Once we have final validation on that build, we will push the package to NuGet. My expectation is that it will take less than a week.

We had a delay in progress and communication, because we had to convince shiproom, why this is the best and only fix.
On top of that we are preparing a plan (based on shiproom feedback) to stop shipping this out-of-ban package entirely in .NET Standard 2.0 and forward all functionality to in-box Desktop framework (.NET Core functionality will be untouched) - i.e. no binding redirects will be needed if you target .NET Standard 2.0. Once I have details on impact on all scenarios, I will update this thread (in 1-2 weeks).

[gulbanana]

That's welcome news, and will make netstandard2.0 easier to use.

[karelz]

@davidsh verified the 4.3.1 package from release branch (warning: myget was very slow for him - 5 mins)
Here are steps to validate:

Please try out the latest packages on the MyGet dev feed. You'll want to use the latest STABLE (not Prerelease) build of System.Net.Http.dll package - 4.3.1:
https://dotnet.myget.org/feed/dotnet-core/package/nuget/System.Net.Http

You can use dev feed packages via changing your NuGet package source feeds either with command line tools or Visual Studio.

Instructions:

Command-Line:
Add the following in nuget.config, under element:

<add key="myget.org dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />

Visual Studio:
In VS, Tools->Options->Nuget Package Manager->Package Sources -> Add, under Source, use this url, https://dotnet.myget.org/F/dotnet-core/api/v3/index.json

In either case, make sure you list the MyGet dev feed as the first feed in the list.

[EDIT]
Expect binding redirect for to 4.1.1.0 in your config file:

<dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
</dependentAssembly>

[karelz]

I have updated top post 'Execution plan' with next steps:

• 5.c Final/last validation of (most of) the ~7 end-to-end scenarios - folks who helped before (or anyone else), can you please reply once you verify with the scenario brief description? Thanks! (we're almost there)
  
  
  
  • cc: @annemartijn0 @karelkrivanek @jahmai @pollax @MikeGoldsmith @chadwackerman @dluxfordhpf @onovotny
  
  
• 5.d Publish the package on myget.org

[davidsh]

Tried to verify but I'm getting errors attempting to install that package.

When I did my validation, I did it with a brand new project.

I suspect that the error you're getting with "Failed to update binding redirects" is due to the downgrade in package and assembly versions. Your current project appears to be based on the packages from the [master] branch. System.Net.Http.4.4.* is package numbering from [master] branch (part of the pre-release for .NET Core 2.0). It has an assembly version for System.Net.Http that is 4.2.*.

The package System.Net.Http version 4.3.1 is a STABLE (not pre-release) package and it is built from the .NET Core 1.1 servicing branch (compatible with .NET Core 1.1.1 servicing release). It contains a binary of System.Net.Http dll with a different assembly version.

I think the bug you've discovered is when Visual Studio/NuGet is trying to re-write your binding redirects for the changed assembly version of System.Net.Http.

So, you could try either to create a fresh new solution/project. Or perhaps delete your binding redirects and then put them back.

[davidsh]

FYI. My log from my package install:

Attempting to gather dependency information for package 'System.Net.Http.4.3.1' with respect to project 'Net46HttpTest3', targeting '.NETFramework,Version=v4.6.1'
Gathering dependency information took 4.27 min
Attempting to resolve dependencies for package 'System.Net.Http.4.3.1' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'System.Net.Http.4.3.1'
Resolved actions to install package 'System.Net.Http.4.3.1'
Retrieving package 'System.Security.Cryptography.Encoding 4.3.0' from 'CoreFx Dev Feed'.
Retrieving package 'System.Security.Cryptography.Primitives 4.3.0' from 'CoreFx Dev Feed'.
Retrieving package 'System.Security.Cryptography.Algorithms 4.3.0' from 'CoreFx Dev Feed'.
Retrieving package 'System.Security.Cryptography.X509Certificates 4.3.0' from 'CoreFx Dev Feed'.
Retrieving package 'System.Net.Http 4.3.1' from 'CoreFx Dev Feed'.
  GET https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.3.1
Adding package 'System.Security.Cryptography.Encoding.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.Encoding.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
  OK https://dotnet.myget.org/F/dotnet-core/api/v2/package/System.Net.Http/4.3.1 302ms
Installing System.Net.Http 4.3.1.
Added package 'System.Security.Cryptography.Encoding.4.3.0' to 'packages.config'
Successfully installed 'System.Security.Cryptography.Encoding 4.3.0' to Net46HttpTest3
Adding package 'System.Security.Cryptography.Primitives.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.Primitives.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.Primitives.4.3.0' to 'packages.config'
Successfully installed 'System.Security.Cryptography.Primitives 4.3.0' to Net46HttpTest3
Adding package 'System.Security.Cryptography.Algorithms.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.Algorithms.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.Algorithms.4.3.0' to 'packages.config'
Successfully installed 'System.Security.Cryptography.Algorithms 4.3.0' to Net46HttpTest3
Adding package 'System.Security.Cryptography.X509Certificates.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.X509Certificates.4.3.0' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Security.Cryptography.X509Certificates.4.3.0' to 'packages.config'
Successfully installed 'System.Security.Cryptography.X509Certificates 4.3.0' to Net46HttpTest3
Adding package 'System.Net.Http.4.3.1' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Net.Http.4.3.1' to folder 'c:\users\dshulman\documents\visual studio 2015\Projects\Net46HttpTest3\packages'
Added package 'System.Net.Http.4.3.1' to 'packages.config'
Successfully installed 'System.Net.Http 4.3.1' to Net46HttpTest3
Executing nuget actions took 2.41 sec
========== Finished ==========
Time Elapsed: 00:06:40.8451462

[pollax]

Hm, I'm confused when testing. I updated to 4.3.1 and got the following binding redirect in my web.config.

<dependentAssembly>
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" />
</dependentAssembly>

Expected?
Maybe I missed something earlier in the thread, or maybe this is one of those confusing packages version vs dll version missmatch situations.
I also uninstalled the package, removed the binding redirects and then reinstalled and I got the same result.

Building and running Works On My Machine™️.

[davidsh]

Incidentally I'm not quite sure why this version dropped from 4.4 to 4.3.1 but okay.

The version dropped in numbering because the 4.4 version will be the latest but it is still pre-release and will shipped as part of .NET Core 2.0. @karelz asked folks to test that package first because the fix was there first.

The 4.3.* packages are based on the RTM .NET Core 1.1. And there will be a servicing release of that. So, the updated package based on that codebase is 4.3.1 for System.Net.Http (since the .NET Core 1.0 package was 4.3.0 for System.Net.Http.

Maybe I missed something earlier in the thread, or maybe this is one of those confusing packages version vs dll version missmatch situations.

Yes, this is confusing. The NuGet package version is not the same as the .NET assembly version of the binary.

For the System.Net.Http 4.3.1 NuGet package, it does contain a binary of System.Net.Http that has a 4.1.1.0 assembly version. So, you are getting the right results.

[karelz]

Thanks @pollax for validation of your end-to-end scenario (top post updated).
Waiting for couple of more validations, before we can ship it as final fix on nuget.org ... almost there ...

My apologies that we missed the binding redirect in instructions (I didn't realize we automatically bump assembly version due to potential GAC'ing, but it makes sense).
Also apologies that myget forces you into all packages from myget -- I am following up with folks internally to find out if we have steps to grab just single package from myget. At least for future verifications.

@davidsh can you please coordinate the end-to-end validations while I am OOF? Once we have ~3 validations, please ask @leecow / @weshaggard to publish the package on nuget.org. Thanks!

[kriscoleman]

Hey guys,

Slightly off topic but I just wanted to give a shoutout to the team here. This issue has been very active and the response has been sometimes hostile. Despite the hostility, I feel the dev team here handled it with class.

Thanks for the support guys and keep up the good work. Mistakes happen. Thanks for fixing it, I understand it takes time.

[Vhab]

Here's another confirmation the new version fixes the issue.

We use KeyVault, bumping to 4.3.1 fixed the issue.

[tofutim]

Hi, I have this problem with SignalR. But how do I get System.Net.Http 4.3.1? I only see 4.3.0 in
https://www.nuget.org/packages/System.Net.Http/

Oops, missed messages about CoreFx - https://dotnet.myget.org/F/dotnet-core/api/v3/index.json

It fixes my SignalR issue.

As noted by others, the feed is super slow. Any chance of getting 4.3.1 into the normal nuget channel? It is Sat at 1:30pm and whoa.... (8 minutes waiting for deps)

---

Attempting to gather dependency information for package 'System.Net.Http.4.3.1' with respect to project 'Translate\Kumquat.Translate.Tests', targeting '.NETFramework,Version=v4.6'
Gathering dependency information took 5.85 min
Attempting to resolve dependencies for package 'System.Net.Http.4.3.1' with DependencyBehavior 'Lowest'
One or more unresolved package dependency constraints detected in the existing packages.config file. All dependency constraints must be resolved to add or update packages. If these packages are being updated this message may be ignored, if not the following error(s) may be blocking the current package operation: 'System.Net.Http 4.3.0'
Resolving dependency information took 0 ms
Resolving actions to install package 'System.Net.Http.4.3.1'
Resolved actions to install package 'System.Net.Http.4.3.1'

Attempting to gather dependency information for package 'System.Net.Http.4.3.1' with respect to project 'Dict\Kumquat.Dict.CE.Tests', targeting '.NETFramework,Version=v4.6'
Gathering dependency information took 3.84 min
Attempting to resolve dependencies for package 'System.Net.Http.4.3.1' with DependencyBehavior 'Lowest'
One or more unresolved package dependency constraints detected in the existing packages.config file. All dependency constraints must be resolved to add or update packages. If these packages are being updated this message may be ignored, if not the following error(s) may be blocking the current package operation: 'System.Net.Http 4.3.0'
Resolving dependency information took 0 ms
Resolving actions to install package 'System.Net.Http.4.3.1'
Resolved actions to install package 'System.Net.Http.4.3.1'

[karelkrivanek]

@karelz Azure Storage API scenario revalidated with the 4.3.1 version from MyGet.

Sorry for not responding sooner.

[maartenba]

@tofutim gathering dependencies is slow because of a lot of metadata going over the wire - https://github.com/NuGet/Home/issues/4448

[tofutim]

I figured that. Any ETA on getting it into nuget.org?

@davidsh Hi David, will this be the week 4.3.1 makes it to nuget? I have a fairly complex project and it feels like wait time must scale with the number of packages in the project. Still, having a fix is better than nothing. I suppose I can copy the nupkg somewhere.

Attempting to gather dependency information for package 'Kumquat.Translate.8.6.2' with respect to project 'qi', targeting '.NETFramework,Version=v4.6'
Gathering dependency information took 8.76 min

Most recently 8.76 min.

[JoeGordonMVP]

@davidsh This just popped up on the OwlMQ mailing list. I can report that the updated package solves it.

Thanks so much to the team for getting ahead of this. Awesome work!

[davidsh]

Thank you for all the validations of the package.

The System.Net.Http 4.3.1 package has been promoted to NuGet.org.

https://www.nuget.org/packages/System.Net.Http/4.3.1

[georgiosd]

Hm, very bizarre - RavenDB client now complains it can't find the 4.1.1 assembly

EDIT: Caveat - Acme.Core references RavenDB.Client and Acme.Main references Acme.Core. VS2015 won't copy System.Net.Http as dependency but binding redirect is there -> boom. Is this expected behavior? Simple enough fix of course...

[karelz]

Closing the issue as fixed. Thanks @davidsh and @CIPop for their work here!
Thanks everyone for their patience. And our apologies for the delay. Next step: post-mortem (as promised) - please give me ~2 weeks to find out all historical details around here ...

@georgiosd can you please file a new issue and provide a repro? (ideally starting with new project)

[tofutim]

@karelz thank you!

[karelz]

FYI:

• I updated top most post with list of verified scenarios (just in case it is ever needed in future) and link to the package.
• For future, I plan to look at steps to use just particular package from myget instead of the entire feed, to workaround the problem of getting everything as latest (and also the slowness issues). Let's hope we won't need it that soon.

[PureKrome]

@karelz quick question: where will we find info about the post-mortem, when that is completed. In this thread _or_ another thread/place?

[karelz]

@PureKrome I will definitely update this thread - it's easier for everyone already interested to get the notification. Also my goal is not to downplay the post-mortem and hide it from people.
I will most likely create a new issue for the discussion though (as I expect some) ;-).
On high level, I plan to cover:

1. How did the issue slip through into release? How to prevent such situation in future?
2. Why did it take 6 months to fix? Why wasn't it treat/communicated as high-impact issue earlier? How to recognize and react to high-impact issues earlier in future?
3. Other concerns (e.g. overall communication)

Also, if we find something broken / not working correctly with 4.3.1 (e.g. @georgiosd's find above), we should mention it here for awareness, but take details into separate issue for easier discussion/follow up.

[PureKrome]

Thank you @karelz (and MS Team). I'll stay subscribed to this thread then. 👍

Keep fighting the good fight, team! 💯

[tofutim]

I also have to thank @chadwackerman2

[georgiosd]

Congrats to all once again for solving one of the most annoying bugs in the history of .NET :)

@karelz happy to do so but I wondered if this is somehow the expected behavior?

[karelz]

@georgiosd I don't know - it will be easier to discuss it in separate issue ;-) (incl. pulling in the right experts)

[gulbanana]

Thanks for driving this work, @karelz

[karelz]

I know I am late with the post-mortem here, my apologies.
I didn't forget, I just didn't get to it yet due to higher priorities (planning/tracking 2.0, looking deeper into the binding redirects issue which surfaced here as well - dotnet/runtime#17770)
I'll try hard to finish the post-mortem in next 1-2 weeks.

[dmcweeney]

Hi, Well done all involved in resolving this. Cant say I understand all the nuts and bolts but one thing I still don't understand is why this only happened when we upgraded our solution from VS 2015 Update 3 to VS 2017. Solution in question was a set of ASP.Net Core projects targeting the .net framework 4.6. Problem was getting triggered by our use of Azure KeyVaultClient.

Thanks, Donal

[parismiguel]

Hi @karelz , I've been researching and working two nights in a row so far but I'm not able to fix this. All my System.Net.Http packages have been updated to version 4.3.1 but still get the same error. Please help me, I don't know what else to try or where to go. Thanks!

FileNotFoundException: Could not load file or assembly 'System.Net.Http, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

An here my .csproj

netcoreapp1.1

$(PackageTargetFallback);portable-net45+win8+wp8+wpa81;

aspnet-IbmVcaCore-5aa05652-04e7-4a42-9fd6-8dbc1c8b98fe

[karelz]

@parismiguel sorry to hear that :(. Do you have bindingRedirects from 4.0.0.0 to 4.1.1.0 in your app?

<dependentAssembly> 
    <assemblyIdentity name="System.Net.Http" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" /> 
    <bindingRedirect oldVersion="0.0.0.0-4.1.1.0" newVersion="4.1.1.0" /> 
</dependentAssembly> 

We're looking into how to make the bindingRedirects story smoother: https://github.com/dotnet/corefx/issues/9846#issuecomment-287941109

UPDATE: Looking at your error, it seems that the problem is that your 4.1.1.0 version was not found. Can you check your app directory if the right assembly version is truly there?

[parismiguel]

@karelz that was fast, thank you very much! I've saw the bindingRedirects stuff in previous posts but I'm not sure where to place it... I've tried inside my .csproj but I've got errors (attached as .txt) Concerning version 4.1.1.0 I'm not sure to follow... Am I not supposed to upgrade it to 4.3.1 ?

IbmVcaCore.csproj.txt

[davidsh]

Binding redirects go into your app.config or web.config file:

[davidsh]

See this for details:
https://msdn.microsoft.com/en-us/library/7wd6ex19(v=vs.110).aspx

[parismiguel]

Hi @davidsh , thank you for your help!
My project doesn't have any of those files... its a NET Core project build using Visul Studio 2017 template... what am I missing?

[karelz]

Re: 4.3.1 vs. 4.1.1 - 4.3.1 is NuGet package version, 4.1.1.0 is assembly version (ildasm/ILSpy/VS will show it to you).
The versions NuGet vs. assembly are a bit confusing and it's hard to connect which belongs to which. It's on my list of things to look deeper into if we can connect the dots via documentation (e.g. on NuGet page).

[karelz]

If you're on .NET Core, you don't need the bindingRedirects and moreover, this issue does NOT affect you at all. This issue is specific to Desktop (= .NET Framework).
The 4.3.1 NuGet package modified only its Desktop assembly, not .NET Core assembly.

[karelz]

If you still have trouble, please file a new bug and tag me there. This issue is blasted to quite a lot folks (as it was impactfull), your problem is unrelated to it (at least it looks like that), so let's be kind to everyone's GH notifications / inboxes.

[karelz]

To everyone: I have created a new issue dotnet/corefx#17522 to track the post-mortem I promised.
Sadly, I didn't make much progress towards it yet :( ... but at least everyone interested can start following that issue and avoid potential noise on this issue (trying to help people focus on what they are interested in).

[parismiguel]

@karelz thanks, I'll follow your instructions and post on your brand new issue tracker. Regards.

[karelz]

@parismiguel not on the new issue I created, please create a brand new one. The one I created (#17522) is for post-mortem analysis why this issue (#11100) took so long to resolve.

[parismiguel]

thank you @karelz and my sincere apologies for messing this topic.... hope to get some help in the new one as instructed. Warm regards,

[Workshop2]

I have seemed to have encountered this via a perfect storm.

I am using the Azure Functions preview with a combination of both Azure Key Vault (2.0.6) and Octopus Client (4.15.3).

I have tried upgrading to System.Net.Http to 4.3.2 however it still fails when trying to use Key Vault.

Any tips?

@karelz

[karelz]

Can you please make sure that the assembly from 4.3.2 nuget package is really used at runtime? (it was fixed in 4.3.1)