Aspnetcore: Kestrel startup issue - address already in use

[Petermarcu]

@istvanko filed this here: https://github.com/dotnet/core/issues/1062

We host some microservices in Azure using ASP.Net Core 2.0 framework and as we noticed, after some days, the services are recycled automatically although Always on is turned on, but the restart doesn't work all the time and our services remain stopped.

We enabled the logging in Azure and we found the below log entry:

2017-11-06 23:27:28.337 +00:00 [Critical] Microsoft.AspNetCore.Server.Kestrel: **Unable to start Kestrel.** System.IO.IOException: **Failed to bind to address http://127.0.0.1:3119: address already in use.** ---> Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.AddressInUseException: Error -4091 EADDRINUSE address already in use ---> Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4091 EADDRINUSE address already in use at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.ThrowError(Int32 statusCode) at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.tcp_getsockname(UvTcpHandle handle, SockAddr& addr, Int32& namelen) at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvTcpHandle.GetSockIPEndPoint() at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.ListenTcp(Boolean useFileHandle) at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.CreateListenSocket() at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.<>c.<StartAsync>b__8_0(Listener listener) at 'Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.CallbackAdapter'1.<>c.<.cctor>b__3_1(Object callback, Object state) at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.DoPostWork() --- End of stack trace from previous location where exception was thrown ---

Detailed stack trace available in the document:
69a2cc3935e9dc939ab7e2d17b47888fef8e50149a09de2203f3b52f2432e924_applicationLog.txt

The issue is quite annoying since it happens in the production system once every week.

We noticed another issue last week, when the dotnet command received invalid arguments which might have connection with this issue.

In general these issues happen in case of Azure infrastructure maintenance or in case of remote file storage unavailability, etc.

[Tratcher]

@pan-wang @shirhatti

[mfahlen]

We are experiencing the same problem in our Azure hosted ASP.NET Core apps.

[halter73]

I took a second look at the stack trace in the original report, and saw there was no Program.Main() up the stack or any indication that the binding failure crashed the process.

The ANCM IIS module which hosts ASP.NET Core apps in the Azure App Service environment will start the dotnet process telling it to listen on a randomly specified port. There is no guarantee that the port that ANCM specified isn't already in use.

If the port is in use, ANCM expects the dotnet process to crash/exit so it can restart the dotnet process with a new random port to listen on. If the process doesn't crash, the dotnet process won't be restarted and the app will be unresponsive since there is no communication between ANCM and the app.

I would double check to ensure any exceptions thrown from WebHost.Run() don't get caught but instead crash the process.

[istvanko]

This issue happens quite frequently. We don't handle exceptions in the Program.cs and in the Startup.cs and there is no custom code on the stack when this happens. Please let me know if you found anything in the framework or there is something we could add to our custom code to crash the application. Maybe on the ANCM side it would be a good idea to replace the random algorithm and check the ports used by IIS. @halter73

[otryshko]

@halter73 we just hit it as well. We use UseExceptionHandler to log exception and return 500 to the user - does it count as "not-crashing"? If yes, what would be the suggested change in UseExceptionHandler lambda

if (exception is System.IO.IOException) throw exception;

or

if (exception is System.IO.IOException) {await next(); return; }

?

[Tratcher]

UseExceptionHandler applies to request processing exceptions, not startup exceptions.

[otryshko]

@Tratcher Thanks! Original question still stands then - our .net core app deployed to Azure suddenly died and we had to restart it to bring it back :(

[istvanko]

@Tratcher, this is actually causing a serious production issue in Azure. It would be great to have a solution soon.

[Tratcher]

Then you should contact Microsoft Support. Our ability to debug your application over github is extremely limited.

[otryshko]

Quick clarification: our callstack is a bit different.

I also opened a case with Microsoft support, let's see how it unravels.

Callstack:
System.IO.IOException: Failed to bind to address http://localhost:1146 on the IPv4 loopback interface: port already in use. ---> System.AggregateException: One or more errors occurred. (Error -4091 EADDRINUSE address already in use) ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4091 EADDRINUSE address already in use
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_getsockname(UvTcpHandle handle, SockAddr& addr, Int32& namelen)
at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.GetSockIPEndPoint()
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListener.CreateListenSocket()
at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener.b__8_0(Object state)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.CreateServer(ServerAddress address)
at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.StartTContext

[otryshko]

Update: as a workaround, I've enabled "auto-healing" via Kudu console:
https://blogs.msdn.microsoft.com/waws/2015/11/04/auto-heal-your-azure-web-app/
Let's see if it helps.

[ThomasArdal]

I'm experiencing this as well. Getting the following exception

System.IO.IOException: Failed to bind to address http://127.0.0.1:3278: address already in use. ---> Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.AddressInUseException: Error -4091 EADDRINUSE address already in use ---> Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4091 EADDRINUSE address already in use
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.ThrowError(Int32 statusCode)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.tcp_getsockname(UvTcpHandle handle, SockAddr& addr, Int32& namelen)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvTcpHandle.GetSockIPEndPoint()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.ListenTcp(Boolean useFileHandle)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.CreateListenSocket()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Listener.<>c.b__8_0(Listener listener)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.CallbackAdapter`1.<>c.<.cctor>b__3_1(Object callback, Object state)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.DoPostWork()

_Always On_ is set to _On_ and I don't have any try-catch in Program.cs or Startup.cs.

Reloading the web app through the portal fixes the problem, but I don't want to do this manually.

[dwdickens]

I just had this occur. Guessing MS is rolling out updates to Meltdown and Spectre which caused a restart of the Azure web app.

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to start Kestrel.
System.IO.IOException: Failed to bind to address http://localhost:10243 on the IPv4 loopback interface: port already in use. ---> System.AggregateException: One or more errors occurred. (Error -4091 EADDRINUSE address already in use) ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4091 EADDRINUSE address already in use
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.tcp_getsockname(UvTcpHandle handle, SockAddr& addr, Int32& namelen)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvTcpHandle.GetSockIPEndPoint()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.TcpListener.CreateListenSocket()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Listener.<StartAsync>b__8_0(Object state)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelEngine.CreateServer(ServerAddress address)
   at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Start[TContext](IHttpApplication`1 application)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Start[TContext](IHttpApplication`1 application)

[halter73]

@dwdickens Did this require a manual restart? If so, can you show us what your Program.Main method looks like? Thanks.

[Eilon]

@halter73 - should this bug move to the Kestrel repo?

[halter73]

This issue was moved to aspnet/KestrelHttpServer#2250