Kestrelhttpserver: Kestrel multi-loop not working on Azure Web Apps

[Tratcher]

_From @esargent on January 13, 2016 22:23_

I got my app published to Azure but I'm getting that dreaded 502.3 Bad Gateway error. However after reading through everything I can find, which mostly relates to beta8/rc1 I think I'm having a slightly different problem where my app is starting just fine, even serving static files (apparently from stdout).
Everything I read here about this says this means you have the wrong version of the IISPlatformHandler, installed on server, but since I'm publishing to Azure is that even something I control? Using rc2-16069 of the Microsoft.aspnet.IISPlaformHandler package. a hopefully relevant snippet of stdout below:

info: Microsoft.Data.Entity.Storage.Internal.RelationalCommandBuilderFactory[1]
      Executed DbCommand (1ms) [Parameters=[@__normalizedName_0='?'], CommandType='Text', CommandTimeout='30']
      SELECT TOP(1) [r].[Id], [r].[ConcurrencyStamp], [r].[Name], [r].[NormalizedName]
      FROM [AspNetRoles] AS [r]
      WHERE [r].[NormalizedName] = @__normalizedName_0
info: Microsoft.Data.Entity.Storage.Internal.RelationalCommandBuilderFactory[1]
      Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
      SELECT COUNT(*)
      FROM [AspNetRoles] AS [a]
Hosting environment: Development
Now listening on: http://localhost:31949
Application started. Press Ctrl+C to shut down.
fail: Microsoft.AspNet.Server.Kestrel[0]
      DispatchPipe.Accept
      Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -4050 ENOTSOCK socket operation on non-socket
         at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check(Int32 statusCode)
         at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.accept(UvStreamHandle server, UvStreamHandle client)
         at Microsoft.AspNet.Server.Kestrel.Http.ListenerSecondary.ReadStartCallback(UvStreamHandle handle, Int32 status)
fail: Microsoft.AspNet.Server.Kestrel[0]
      DispatchPipe.Accept
      Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -4050 ENOTSOCK socket operation on non-socket
         at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check(Int32 statusCode)
         at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.accept(UvStreamHandle server, UvStreamHandle client)
         at Microsoft.AspNet.Server.Kestrel.Http.ListenerSecondary.ReadStartCallback(UvStreamHandle handle, Int32 status)
info: Microsoft.AspNet.Hosting.Internal.WebApplication[1]
      Request starting HTTP/1.1 GET http://aldehydes.azurewebsites.net/  
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
      Executing action method Revelar.Controllers.HomeController.Index with arguments () - ModelState is Valid'
info: Microsoft.AspNet.Mvc.ViewFeatures.ViewResultExecutor[1]
      Executing ViewResult, running view at path /Views/Home/Index.cshtml.
info: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]
      Executed action Revelar.Controllers.HomeController.Index in 0.2234ms
info: Microsoft.AspNet.Hosting.Internal.WebApplication[2]
      Request finished in 3484ms 200 text/html; charset=utf-8
info: Microsoft.AspNet.Hosting.Internal.WebApplication[1]
      Request starting HTTP/1.1 GET http://aldehydes.azurewebsites.net/css/fonts.css  
info: Microsoft.AspNet.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/css/fonts.css'. Physical path: 'D:\home\site\wwwroot\css\fonts.css'
info: Microsoft.AspNet.Hosting.Internal.WebApplication[2]
      Request finished in 47ms 200 text/css

I've tried a couple changes to web.config from what I've read but it is basically:

<configuration>
  <system.webServer>
    <handlers>
      <remove name="httpplatformhandler" />
      <add name="httpplatformhandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified" />
    </handlers>
    <httpPlatform processPath="%home%\site\approot\web.cmd" arguments="" stdoutLogEnabled="true" stdoutLogFile="\\?\%home%\LogFiles\stdout.log" forwardWindowsAuthToken="false" startupTimeLimit="3600"></httpPlatform>
  </system.webServer>
</configuration>

_Copied from original issue: aspnet/IISIntegration#64_

[Tratcher]

Error -4050 ENOTSOCK

@halter73, @cesarbs?

[Tratcher]

_From @halter73 on January 14, 2016 1:28_

As a workaround, you can probably do something like:

// ...
using Microsoft.AspNet.Http.Features;
using Microsoft.AspNet.Server.Kestrel;
// ...

public class Startup
{
    public void Configure(IPplicationBuilder app, ...)
    {
        var ksi = app.ServerFeatures.Get<IKestrelServerInformation>();
        ksi.ThreadCount = 1;
        // ...

I'll be interested to see if that fixes things. If it does, then Error -4050 ENOTSOCK is likely relevant.

Everything _should_ still work with a ThreadCount > 1. I'll have to investigate further.

[Tratcher]

_From @esargent on January 14, 2016 17:53_

That fixed it. Can't wait to understand more what was happening!!

[DamianEdwards]

Needed for RC2.

I'm guessing this only repros when hosting on a Large or bigger (4+ cores) as our default logic is to do CPU count / 2.

[esargent]

Not sure what is going on underneath, but this was with the F1 Free dev tier
https://azure.microsoft.com/en-us/pricing/details/app-service/

[DamianEdwards]

Interesting. I wonder what the CPU count is on those shared tiers.

[halter73]

@esargent I'm not sure what it is. I think I might have seen this repro on the free tier too when I was investigating, but not usually. By default, ThreadCount is determined using Environment.ProcessorCount >> 1, and this bug obviously doesn't repro when the ThreadCount is 1.

My guess is that Environment.ProcessorCount isn't consistent on the free tier.

[esargent]

@halter73 I suspect you guys are in a better position to find out from Azure what the expectations are for those enviornments, but if you want me to try/test/do anything just let me know.

And, yes, confirming I've not seen any issues since adding the hack you recommended last week to force thread count to 1.

[muratg]

cc @moozzyk

[cesarblum]

We have identified this is an Azure sandbox issue. It can be reproduced by creating a socket, duplicating it and setting the duplicate to be non-blocking:

var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
var socketInfo = socket.DuplicateAndClose(Process.GetCurrentProcess().Id);
var duplicate = new Socket(socketInfo);
duplicate.Blocking = false;

We're waiting for an answer from Azure folks.

[muratg]

We'll close once we verify Azure fix.

[cesarblum]

Azure fix should be available next week.

[cesarblum]

@esargent Which region is your app deployed to?

[esargent]

The app is in US West currently. Will want to deploy to Ireland for production, but that's a few months out.

[Eilon]

@muratg @cesarbs any work remaining for us on this?

[muratg]

@Eilon No, the issue is fixed, but it's taking it's time to propagate across Azure regions. I think @cesarbs kept these open until all regions has the fix. I'm OK with both -- keeping it open or closing it.

[Eilon]

Ok so all we need to do is verify it? Perhaps close this bug, and log a new bug saying we need to verify it? That would just be a 'task' item.

[muratg]

@cesarbs did the verification in one of the regions, and it's fixed. I think the reason we didn't close it yet is to tell customers to wait until it's fixed for their data centers. But since the issue is essentially fixed, we can close this one. We'll respond to folks who ask about this and tell them to wait or use another region :)

[Eilon]

Sounds good!

[cesarblum]

@esargent I've just verified my test web app for this issue and it looks like it's fixed in West US. Can you check that you don't see the ENOTSOCK errors anymore when running without the ThreadCount = 1 workaround?

[esargent]

I just deployed successfully after removing the code to set threadcount. fix verified.
Thank you!