Roslyn: Long-running VBCSCompiler.dll process runs at 100% CPU after parent process dies

[natemcmaster]

Version Used:
Using dotnet SDK 2.2.0-preview1-007866 on macOS 10.13.2

Steps to Reproduce:
TBH not sure 100% how to get into bad state. I don't know what is launching VBCSCompiler.dll. The parent process is gone...which is the issue.

But, it has been happening fairly often over the last few days while working on the https://github.com/dotnet/cli repo using VS Code - Insiders + C# extension 1.13.1.

If someone can point me to how to capture more information about how this long-running compiler server starts, I can provide a better repro.

Expected Behavior:
No long running compiler process after I shutdown tools and build scripts.

Actual Behavior:

VS Code is closed, and I've killed the the build.sh script, yet I have two VBCSCompiler processes running. ps reports the processes have been open for ~ 18 hours.

The process was launched with these arguments:

/Users/namc/.dotnet/dotnet /Users/namc/.dotnet/sdk/2.2.0-preview1-007866/Roslyn/bincore/VBCSCompiler.dll -pipename:namc.False.bZ9mpL2HFmG8IVwxPkEmz6_N4OMYW1cLZOeZGRjGfv0

Process sample:
processample.txt

[EDIT]

See below for a workaround: https://github.com/dotnet/roslyn/issues/24137#issuecomment-388494024

[jkotas]

Here is my analysis of dump that @natemcmaster shared offline:

Main thread in the dump is inside the “close socket” syscall. The full stack of the main thread is below. My guess is that the main thread is looping at this stack. No other threads are doing anything interesting.

The interesting frame is the one handling exception in System.IO.Pipes.NamedPipeServerStream+SharedServer..ctor at https://github.com/dotnet/corefx/blob/master/src/System.IO.Pipes/src/System/IO/Pipes/NamedPipeServerStream.Unix.cs#L296 .

The exception being handled is ArgumentOutOfRangeException. The error message in the exception being handled is: u"The path '/var/folders/2m/hdplh2w93rbgy0yqd6hcmcq00000gp/T/CoreFxPipe_namc.False.Ve_W9KuAp4xXKB90AGmatCAaOjEjd0fExSPR472pVQI' is of an invalid length for use with domain sockets on this platform. The length must be between 1 and 104 characters, inclusive."

It sounds like an implementation limitation or bug in System.IO.Pipes to me.

 * frame #0: 0x00007fff7438d7d2 libsystem_kernel.dylib`close + 10
    frame #1: 0x000000011cdd5f4f DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr)
    frame #2: 0x000000011d96f2a5 System.Net.Sockets.SafeCloseSocket+InnerSafeCloseSocket.InnerReleaseHandle()
    frame #3: 0x000000011d96f1b9 System.Net.Sockets.SafeCloseSocket+InnerSafeCloseSocket.ReleaseHandle()
    frame #4: 0x000000010e1254e7 libcoreclr.dylib`CallDescrWorkerInternal + 124
    frame #5: 0x000000010df94d97 libcoreclr.dylib`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 215
    frame #6: 0x000000010e05d6f7 libcoreclr.dylib`SafeHandle::RunReleaseMethod(SafeHandle*) + 135
    frame #7: 0x000000010e05d5f2 libcoreclr.dylib`SafeHandle::Release(bool) + 642
    frame #8: 0x000000010e05e428 libcoreclr.dylib`SafeHandle::DangerousRelease(SafeHandle*) + 168
    frame #9: 0x000000011d96eafe System.Net.Sockets.SafeCloseSocket.CloseAsIs()
    frame #10: 0x000000011d95efcb System.Net.Sockets.Socket.Dispose(Boolean)
    frame #11: 0x000000011d95f47f System.Net.Sockets.Socket.Dispose()
    frame #12: 0x000000011d841dda System.IO.Pipes.NamedPipeServerStream+SharedServer..ctor(System.String, Int32)
    frame #13: 0x000000011d841bb5 System.IO.Pipes.NamedPipeServerStream+SharedServer.Get(System.String, Int32)
    frame #14: 0x000000011d84129f System.IO.Pipes.NamedPipeServerStream.Create(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode, System.IO.Pipes.PipeOptions, Int32, Int32, System.IO.HandleInheritability)
    frame #15: 0x000000011d840d4a System.IO.Pipes.NamedPipeServerStream..ctor(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode, System.IO.Pipes.PipeOptions, Int32, Int32, System.IO.HandleInheritability)
    frame #16: 0x000000011d840c80 System.IO.Pipes.NamedPipeServerStream..ctor(System.String, System.IO.Pipes.PipeDirection, Int32, System.IO.Pipes.PipeTransmissionMode, System.IO.Pipes.PipeOptions, Int32, Int32)
    frame #17: 0x000000011cea1e28 Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost+<CreateListenTaskCore>d__6.MoveNext()
    frame #18: 0x000000011cdd40a9 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost+<CreateListenTaskCore>d__6, VBCSCompiler]](<CreateListenTaskCore>d__6 ByRef)
    frame #19: 0x000000011ce9ac02 Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost.CreateListenTaskCore(System.Threading.CancellationToken)
    frame #20: 0x000000011cea1bc3 Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost+<CreateListenTask>d__5.MoveNext()
    frame #21: 0x000000011cdd3989 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost+<CreateListenTask>d__5, VBCSCompiler]](<CreateListenTask>d__5 ByRef)
    frame #22: 0x000000011ce9ab82 Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost.CreateListenTask(System.Threading.CancellationToken)
    frame #23: 0x000000011ce9d62e Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.CreateListenTask()
    frame #24: 0x000000011ce9d755 Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.HandleCompletedListenTask(System.Threading.CancellationToken)
    frame #25: 0x000000011ce9d2da Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.CheckCompletedTasks(System.Threading.CancellationToken)
    frame #26: 0x000000011ce9d23c Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.ListenAndDispatchConnectionsCore(System.Threading.CancellationToken)
    frame #27: 0x000000011ce9d0b8 Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.ListenAndDispatchConnections(System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
    frame #28: 0x000000011ce9cc60 Microsoft.CodeAnalysis.CompilerServer.BuildServerController.RunServerCore(System.String, Microsoft.CodeAnalysis.CompilerServer.IClientConnectionHost, Microsoft.CodeAnalysis.CompilerServer.IDiagnosticListener, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
    frame #29: 0x000000011ce9a6db Microsoft.CodeAnalysis.CompilerServer.DesktopBuildServerController.RunServerCore(System.String, Microsoft.CodeAnalysis.CompilerServer.IClientConnectionHost, Microsoft.CodeAnalysis.CompilerServer.IDiagnosticListener, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
    frame #30: 0x000000011ce9cda9 Microsoft.CodeAnalysis.CompilerServer.BuildServerController.RunServer(System.String, Microsoft.CodeAnalysis.CompilerServer.IClientConnectionHost, Microsoft.CodeAnalysis.CompilerServer.IDiagnosticListener, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
    frame #31: 0x000000011ce9ca83 Microsoft.CodeAnalysis.CompilerServer.BuildServerController.Run(System.String[])
    frame #32: 0x000000011ce9b0b6 Microsoft.CodeAnalysis.CompilerServer.VBCSCompiler.Main(System.String[])

[jcouv]

Maybe related to https://github.com/dotnet/roslyn/issues/24216 (VBCSCompiler running at high CPU on Mac)

[dasMulli]

Also observing this consistently on 4 Macs (high CPU after a build is done).

While running the integration tests of dotnet/sdk, four vbcscompiler instances are started leading to full (!) cpu usage on my quad-core macbook.

[jaredpar]

CC @agocke

This looks like another case where the limitations on domain socket name lengths may be biting us. Where are we with the mitigations there.

[agocke]

@jaredpar Looks like I created a commit locally, but it didn't get included with the PR for the other fix. Creating a new PR now.

[halter73]

@natemcmaster @agocke Do either of you know what version of the dotnet SDK was fixed in? I'm still seeing this issue when building Kestrel's dev branch on a mac.

shalter@halter73-mbp Kestrel.Transport.Libuv.FunctionalTests (dev$ u=)$ dotnet --info
.NET Command Line Tools (2.2.0-preview1-008003)

Product Information:
 Version:            2.2.0-preview1-008003
 Commit SHA-1 hash:  839eccbcaf

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /Users/shalter/.dotnet/sdk/2.2.0-preview1-008003/

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview2-26130-06
  Build    : b13a0d5c331f374afd35ded57b9a4b4ab128864c

shalter@halter73-mbp Kestrel.Transport.Libuv.FunctionalTests (dev$ u=)$ ps aux | grep dotnet
shalter          16604 106.9  2.6 24361308 429404 s004  R     1:31PM   1:53.07 /Users/shalter/.dotnet/dotnet /Users/shalter/.dotnet/sdk/2.2.0-preview1-008003/Roslyn/bincore/VBCSCompiler.dll -pipename:shalter.False.PeduO9n_a+GQj5YqfeYoNiQq_D3ri1GYtBAP3fgC1uE

[natemcmaster]

@halter73 try updating kestrel to the latest build tools by running build.sh -u. We upgraded the SDK in build tools a few days ago, but that change won't propagate automatically to kestrel until Sunday.

[halter73]

@natemcmaster That did the trick. Thanks.

[cahlbin]

@natemcmaster Do you know if the fix for this is merged into any currently public available .NET SDK release (possibly preview)? I'm now running 2.1.300-preview2-008533 and observing this issue. I can't seem to find any 2.2.x (as you referred @halter73 to (?)) (not even preview) on the official .NET downloads pages.

$ dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.1.300-preview2-008533
 Commit:    996eb6c92c

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.13-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.1.300-preview2-008533/

Host (useful for support):
  Version: 2.1.0-preview2-26406-04
  Commit:  6833f3026b

Other:

system username length: 18 chars (seem like that can contribute to whether this issue occurs or not)

process-sample.txt

[natemcmaster]

The 2.1.300-preview2 SDK should have this fix. This may have resurfaced since, or this is a new bug with similar symptoms.

@agocke any ideas here?

[agocke]

Sorry, missed this notification earlier. I also thought this bug was resolved.

@cahlbin Can you try running a build with the environment variable RoslynCommandLineLogFile=<path> set? If that works there should be a text file at <path> that contains a log for the compiler server that may help in tracking down the issue.

[mikeharder]

I'm still seeing this on my Mac with 2.1.300-rc1-008673. The log shows this exception being thrown in a tight loop:

--- PID=6894 TID=2 Ticks=9242688: Constructing pipe 'mhardermicrosoft.F.7rupEzOZmwqFpJFqR9kpfAHyL'.
--- PID=6894 TID=2 Ticks=9242688: Successfully constructed pipe 'mhardermicrosoft.F.7rupEzOZmwqFpJFqR9kpfAHyL'.
--- PID=6894 TID=2 Ticks=9242688: Waiting for new connection
--- PID=6894 TID=2 Ticks=9242689: Exception 'The path '/var/folders/f0/fbv8dr2x1354d26n9k2wdq700000gn/T/CoreFxPipe_mhardermicrosoft.F.7rupEzOZmwqFpJFqR9kpfAHyL' is of an invalid length for use with domain sockets on this platform.  The length must be between 1 and 103 characters, inclusive.
Parameter name: path
Actual value was /var/folders/f0/fbv8dr2x1354d26n9k2wdq700000gn/T/CoreFxPipe_mhardermicrosoft.F.7rupEzOZmwqFpJFqR9kpfAHyL.' occurred during 'Error creating client named pipe'. Stack trace:
   at System.Net.Sockets.UnixDomainSocketEndPoint..ctor(String path)
   at System.IO.Pipes.NamedPipeServerStream.<WaitForConnectionAsyncCore>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost.<CreateListenTaskCore>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.CompilerServer.NamedPipeClientConnectionHost.<CreateListenTask>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.CodeAnalysis.CompilerServer.ServerDispatcher.<HandleClientConnection>d__28.MoveNext()

A factor might be that my username is relatively long ("mhardermicrosoft" = 16 chars).

[agocke]

Yup, same issue. We can try to strip down the path hash even more, but unless we want to drop the username it's hard to get it much smaller

[mikeharder]

I was able to workaround this issue by creating the file ~/Library/LaunchAgents/UseSharedCompilation.plist with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>UseSharedCompilation</string>
  <key>ProgramArguments</key>
  <array>
    <string>sh</string>
    <string>-c</string>
    <string>launchctl setenv UseSharedCompilation false</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Then logout and login. This should disable the shared compiler server for all apps, including Terminal, Visual Studio Code, and Visual Studio for Mac.

The workaround should be compatible with OS X Yosemite (10.10) or later.

[natemcmaster]

@mikeharder did you test if this solves the issue for VSCode and Visual Studio for Mac? I'm fuzzy on the details, but IIRC apps launched from Finder (i.e. from the Dock or Spotlight) won't have environment variables set as defined in ~/.profile.

[mikeharder]

@natemcmaster: Good catch. I've updated my workaround to use ~/Library/LaunchAgents/UseSharedCompilation.plist rather than ~/.profile, which should set the environment variable for all apps, including those launched from Finder.

[JoiGud]

@mikeharder I tried your workaround and it seems to work. Thank you!

[agocke]

Another workaround: if you set the TMPDIR environment variable to /tmp, that will probably fix the problem and let you still use the compiler server.

[ChrisMcKee]

Similar issues on a long running Teamcity build agent

.NET Core SDK (reflecting any global.json):
 Version:   2.1.402
 Commit:    3599f217f4

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.402/

Host (useful for support):
  Version: 2.1.4
  Commit:  85255dde3e

.NET Core SDKs installed:
  2.0.0 [/usr/share/dotnet/sdk]
  2.1.3 [/usr/share/dotnet/sdk]
  2.1.105 [/usr/share/dotnet/sdk]
  2.1.402 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[danpowell88]

I also have the same issue, even after manually killing all on the next build they all come back

I've also set
DOTNET_RUNNING_IN_CONTAINER=true
TMPDIR=/tmp

We are using teamcity and have also updated to the latest dotnet-cli runner

[dhaval24]

I am still facing this issue. Running on MacOS 10.14.4 on VSCode

[dasMulli]

@dhaval24 what does a dotnet --info print? which version of the C# extension?

[dhaval24]

@dasMulli Here is the output of dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.2.106
 Commit:    aa79b139a8

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.14
 OS Platform: Darwin
 RID:         osx.10.14-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.2.106/

Host (useful for support):
  Version: 2.2.4
  Commit:  f95848e524

.NET Core SDKs installed:
  2.2.106 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

I am running on 1.19.0 C# extension.