Kestrelhttpserver: Unix socket fails with Unexpected exception in FrameConnection.ProcessRequestsAsync

[zpodlovics]

Hi,

I have tried to use Kestrel with unix socket behind Nginx, unfortunately after a successful startup the first unix socket connection will crash it. The same problem are reproducable with curl. The same test url on loopback connection works fine. The example project are attached ( kestrelunixsocketnre.zip ).

Steps to reproduce:

1) cd /tmp
2) unzip path/to/kestrelunixsocketnre.zip
3) cd kestrelunixsocketnre
4) dotnet run
5) curl --no-buffer -XGET --unix-socket /tmp/kestrel-test.sock http://localhost/test
( in an other console, please note this command will hang)

6) press ctrl+c to curl

Expected result:
works without error

Actual result:

Hosting environment: Production
Content root path: /tmp/kestrelunixsocketnre
Now listening on: http://127.0.0.1:5000
Now listening on: http://unix:/tmp/kestrel-test.sock
Application started. Press Ctrl+C to shut down.
fail: Microsoft.AspNetCore.Server.Kestrel[0]
      Unexpected exception in FrameConnection.ProcessRequestsAsync.
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource.ConnectionStart(IConnectionContext context, IConnectionInformation information)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.<ProcessRequestsAsync>d__42`1.MoveNext()
fail: Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv[0]
      UbReadCb
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.Abort(Exception ex)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnection.OnRead(UvStreamHandle handle, Int32 status)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnection.<>c.<.cctor>b__33_0(UvStreamHandle handle, Int32 status, Object state)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvStreamHandle.UvReadCb(IntPtr handle, Int32 status, uv_buf_t& buf)

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.Abort(Exception ex)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnection.OnRead(UvStreamHandle handle, Int32 status)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvConnection.<>c.<.cctor>b__33_0(UvStreamHandle handle, Int32 status, Object state)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvStreamHandle.UvReadCb(IntPtr handle, Int32 status, uv_buf_t& buf)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvStreamHandle.<>c.<.cctor>b__20_2(IntPtr handle, Int32 status, uv_buf_t& buf)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.NativeMethods.uv_run(UvLoopHandle handle, Int32 mode)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.LibuvFunctions.run(UvLoopHandle handle, Int32 mode)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread.ThreadStart(Object parameter)

environment:
Ubuntu 16.04 x86_64

dotnet packages:

ii  dotnet-host                                                 2.0.5-1                                                     amd64        Microsoft .NET Core Host - 2.0.5
ii  dotnet-hostfxr-2.0.5                                        2.0.5-1                                                     amd64        Microsoft .NET Core Host FX Resolver - 2.0.5 2.0.5
ii  dotnet-runtime-2.0.5                                        2.0.5-1                                                     amd64        Microsoft .NET Core Runtime - 2.0.5 Microsoft.NETCore.App 2.0.5
ii  dotnet-sdk-2.1.4                                            2.1.4-1                                                     amd64        Microsoft .NET Core SDK - 2.1.4

curl package:

ii  curl                                                        7.47.0-1ubuntu2.6                                           amd64        command line tool for transferring data with URL syntax

Thank for your help,
Best Regards,
Zoltan

[davidfowl]

Yep this is a bug in Kestrel, we have 0 test coverage for this scenario and need to add it. I think we'll want to patch this in 2.0.x

/cc @halter73 @muratg

[muratg]

@Eilon this is a patch candidate.

[halter73]

@zpodlovics I was unable to reproduce the failure. Here's what I see:

shalter@f4-linux-server:/tmp/kestrelunixsocketnre$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial
shalter@f4-linux-server:/tmp/kestrelunixsocketnre$ dotnet run
Hosting environment: Production
Content root path: /tmp/kestrelunixsocketnre
Now listening on: http://127.0.0.1:5000
Now listening on: http://unix:/tmp/kestrel-test.sock
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost/  
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 103.8834ms 200 
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost/test  
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 1.2366ms 200 

Hello World!shalter@f4-linux-server:~/halter73/kestrelunixsocketnre$ curl --no-buffer -XGET --unix-socket /tmp/kestrel-test.sock http://localhost
Hello World!shalter@f4-linux-server:~/halter73/kestrelunixsocketnre$ curl --no-buffer -XGET --unix-socket /tmp/kestrel-test.sock http://localhost/test

shalter@f4-linux-server:~$ lsof -p 119128
...
dotnet  119128 shalter   15rR     REG                8,1    38912  790461 /home/shalter/.dotnet/shared/Microsoft.NETCore.App/2.0.5/System.Runtime.dll
...
dotnet  119128 shalter   23rR     REG                8,1   273904  600391 /home/shalter/.nuget/packages/microsoft.aspnetcore.server.kestrel.core/2.0.1/lib/netstandard2.0/Microsoft.AspNetCore.Server.Kestrel.Core.dll

Is there anything else I need to do to reproduce the bug? Our theory is that Kestrel's EventSource was enabled when it shouldn't be. Is it possible that you enabled some sort of diagnostic feature that cause this?

[halter73]

We should definitely add tests though. I didn't realize until seeing this issue that curl has a --unix-socket flag. That should make adding a basic linux/travis-only test to ensure a simple request succeeds pretty trivial.

[zpodlovics]

@halter73 You are right, I have environment variables for dotnet core in /etc/profile.d/dotnet.sh

#!/bin/sh
export DOTNET_CLI_TELEMETRY_OPTOUT=1
export COMPlus_PerfMapEnabled=1
export COMPlus_EnableEventLog=1

setting the COMPlus_EnableEventLog variable to 0 solves the exception problem. However disabling this will introduce an another, as it will be no longer possible (yet) to have memory allocation stack trace flamegraph on linux [1] without it.

[1] http://blogs.microsoft.co.il/sasha/2018/02/06/getting-stacks-for-lttng-events-with-net-core-on-linux/

[Eilon]

Doesn't meet the bar for a patch. Removing milestone so this can be reconsidered for 2.1.0.